{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We start with high-precision and high-recall retrieval methods as LightRAG helps you optimize the later stage of your search/retrieval pipeline. As the first stage is often comes with cloud db providers with their search and filter support."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## FAISS retriever (simple)\n",
    "\n",
    "We mainly use this to quickly show how to implement semantic search as retriever."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# decide a meaningful query and a list of documents\n",
    "query_1 = \"What are the benefits of renewable energy?\" # gt is [0, 3]\n",
    "query_2 = \"How do solar panels impact the environment?\" # gt is [1, 2]\n",
    "\n",
    "documents =[\n",
    "    {\n",
    "        \"title\": \"The Impact of Renewable Energy on the Economy\",\n",
    "        \"content\": \"Renewable energy technologies not only help in reducing greenhouse gas emissions but also contribute significantly to the economy by creating jobs in the manufacturing and installation sectors. The growth in renewable energy usage boosts local economies through increased investment in technology and infrastructure.\"\n",
    "    },\n",
    "    {\n",
    "        \"title\": \"Understanding Solar Panels\",\n",
    "        \"content\": \"Solar panels convert sunlight into electricity by allowing photons, or light particles, to knock electrons free from atoms, generating a flow of electricity. Solar panels are a type of renewable energy technology that has been found to have a significant positive effect on the environment by reducing the reliance on fossil fuels.\"\n",
    "    },\n",
    "    {\n",
    "        \"title\": \"Pros and Cons of Solar Energy\",\n",
    "        \"content\": \"While solar energy offers substantial environmental benefits, such as reducing carbon footprints and pollution, it also has downsides. The production of solar panels can lead to hazardous waste, and large solar farms require significant land, which can disrupt local ecosystems.\"\n",
    "    },\n",
    "    {\n",
    "        \"title\":  \"Renewable Energy and Its Effects\",\n",
    "        \"content\": \"Renewable energy sources like wind, solar, and hydro power play a crucial role in combating climate change. They do not produce greenhouse gases during operation, making them essential for sustainable development. However, the initial setup and material sourcing for these technologies can still have environmental impacts.\"\n",
    "    }\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Embedder(\n",
       "  model_kwargs={'model': 'text-embedding-3-small', 'dimensions': 256, 'encoding_format': 'float'}, \n",
       "  (model_client): OpenAIClient()\n",
       ")"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# create an embedder\n",
    "from lightrag.core.embedder import Embedder \n",
    "from lightrag.core.types import ModelClientType\n",
    "\n",
    "\n",
    "model_kwargs = {\n",
    "    \"model\": \"text-embedding-3-small\",\n",
    "    \"dimensions\": 256,\n",
    "    \"encoding_format\": \"float\",\n",
    "}\n",
    "\n",
    "embedder = Embedder(model_client =ModelClientType.OPENAI(), model_kwargs=model_kwargs)\n",
    "embedder"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "256 4 EmbedderOutput(data=[Embedding(embedding=[0.006102133, 0.07962484, 0.14928514, 0.041595064, 0.062026925, 0.04285206, -0.016078092, 0.06906609, 0.05352508, 0.06513513, 0.025711235, 0.015723849, -0.12478519, 0.041229393, 0.04548032, -0.005096538, -0.06028999, 0.0075248214, 0.027653862, 0.06924892, -0.03780123, 0.07505395, 0.005767887, -0.012604219, -0.012112848, 0.027813843, -0.047262963, 0.025574109, 0.054027874, -0.02331152, 0.073225595, -0.030579228, -0.05503347, -0.07971626, 0.13621241, 0.0521081, -0.02326581, -0.0907778, 0.031219153, 0.0018483521, 0.026374014, -0.067237735, -0.10357628, -0.031744804, -0.028408058, -0.012524229, -0.052839443, -0.066186436, 0.07345414, 0.077567935, 0.07724798, -0.081773154, 0.05174243, -0.22360775, -0.03604144, -0.05334224, -0.010495897, 0.048999898, -0.093794584, -0.005770744, 0.16729443, -0.012489947, 0.074094065, 0.12405385, -0.057181787, 0.08625262, -0.011507206, 0.022260215, -0.053707913, -0.0017255095, 0.0026925376, -0.049594115, 0.057136077, -0.07336272, 0.07295135, -0.054896343, 0.05823309, -0.08017334, -0.038349736, 0.040863723, -0.03992669, -0.055399142, -0.05219952, -0.07628809, -0.009490303, 0.0029053695, 0.04207501, -0.01846638, -0.12304825, 0.07706514, -0.019883355, -0.018809197, 0.077613644, 0.029619342, -0.039378185, 0.0052679465, 0.030144995, 0.055262014, 0.045228917, 0.105678886, 0.011427215, 0.014066902, 0.023094403, 0.041686483, 0.06815191, 0.016192364, -0.0062164054, 0.045914553, 0.042646367, -0.015072498, -0.01153006, 0.10403337, 0.0069363196, -0.01007309, 0.046028823, -0.030899191, -0.14114897, -0.0054879203, -0.027059646, -0.048497103, -0.06527226, 0.011770032, 0.032750398, 0.07770506, 0.082550205, 0.07459686, -0.110798284, 0.063672446, -0.061204165, 0.043012038, 0.021597438, 0.08881232, 0.09000075, -0.04026951, -0.054576382, -0.033024654, 0.03419023, -0.04621166, -0.041023705, 0.04420047, 0.03837259, 0.018763488, -0.004336628, 0.02934509, -0.04637164, -0.043446273, 0.013152725, -0.054393545, 0.013815504, 0.07560246, 0.0005299371, 0.012615646, -0.10842142, 0.07103157, -0.040246654, -0.057090368, -0.051559597, -0.010587315, -0.05091967, -0.079304874, -0.071580075, 0.1696713, -0.009353176, -0.051193926, -0.030624937, -0.012947035, -0.03419023, -0.06874613, -0.101382256, -0.050691128, -0.0062278323, -0.12259117, 0.05462209, 0.10458188, -0.03435021, 0.08360151, -0.116923265, 0.06019857, 0.14654261, 0.015655285, -0.07441403, 0.119391546, -0.08533844, -0.030739209, 0.04447472, -0.011701468, -0.08753247, 0.028248077, 0.027265336, 0.10083375, -0.053982165, 0.03181337, -0.0319962, 0.0701631, 0.05462209, 0.025551254, -0.04038378, 0.02060327, -0.0067820526, -0.08730393, 0.029733615, -0.05343366, 0.071397245, -0.05059971, -0.013027025, -0.013358415, -0.08652688, -0.021528874, 0.06856329, -0.016146656, 0.005576481, 0.06330677, -0.09324608, 0.115094915, 0.053753622, 0.11792886, -0.045046084, 0.05128534, 0.029825032, -0.05818738, -0.13602957, -0.06019857, 0.07212859, -0.018614935, 0.027493881, 0.0031424842, -0.0045366045, -0.009467448, 0.09443451, -0.009016073, -0.09228619, -0.032704692, -0.003799549, -0.10284494, 0.01686657, -0.0095474385, -0.0077590793, -0.03604144, -0.057684585, 0.01834068, -0.0083247265, 0.07190004, 0.0060221427, -0.09882256, 0.01325557, 0.064495206, 0.012227121, -0.036475673, 0.12761915, -0.041915026, -0.077430815, -0.006193551, 0.08721251, 0.042052153, 0.11326656, -0.006787766], index=0), Embedding(embedding=[-0.009021607, 0.02116233, -0.051248793, 0.123291686, 0.078064516, 0.023609968, -0.06025957, -0.045660377, 0.13698113, -0.04366761, 0.0066551953, 0.032988973, -0.018021554, 0.023609968, 0.093660094, -0.059263185, -0.013072125, 0.035869822, 0.038447425, 0.05895994, 0.08638216, -0.07238946, 0.025732698, -0.014046848, -0.095912784, 0.011263471, -0.028743511, -0.004830296, 0.12554438, -0.024151482, -0.062468942, -0.033508826, -0.07139308, -0.0018411443, -0.0035442028, 0.10527013, -0.050469015, -0.057963554, -0.005902492, -0.014675003, 0.031516057, -0.0632054, -0.03034639, -0.010727373, 0.009330269, 0.002266232, -0.025061222, -0.09010777, -0.019678583, 0.0350034, 0.057660308, -0.11064194, 0.048389602, -0.020869913, -0.04206473, 0.0062273997, -0.07399234, 0.10527013, -0.151537, 0.025039563, 0.117833234, 0.005274337, 0.031342775, 0.08274319, -0.100504816, 0.08391286, -0.055797502, 0.055017725, -0.040071964, 0.10015825, 0.119826004, 0.0005421899, 0.11506069, -0.015703877, -0.02080493, 0.065111525, 0.0675375, -0.06602127, 0.074598834, 0.07945079, 0.07078658, 0.031451076, -0.110728584, -0.019819377, -0.09227382, 0.014252623, 0.025039563, -0.035869822, -0.009584781, 0.026880706, 0.012021589, -0.000566558, -0.090281054, 0.09972504, -0.043797575, 0.046440158, -0.089154705, -0.0017179501, 0.0012820321, 0.049819198, 0.071739644, 0.020999875, -0.020707458, 0.008133526, 0.05207189, -0.009915103, -0.029891519, 0.0623823, -0.073212564, 0.026079267, -0.02120565, -0.008496339, -0.01745838, 0.030996205, 0.050035805, -0.026620781, -0.034786798, 0.06701765, -0.0126714045, -0.030433033, -0.10189109, 0.020599155, 0.06593463, 0.090281054, 0.10743618, -0.1253711, -0.05614407, -0.0391189, 0.039313845, 0.036108088, 0.07628836, 0.056274034, 0.017036, -0.091407396, -0.06918371, -0.13160932, 0.08304644, -0.06355197, 0.0034710984, 0.049126063, 0.016971018, 0.07979736, -0.053458165, 0.084692635, -0.058006875, -0.09929183, 0.07438223, -0.095046364, -0.029869858, 0.059696395, 0.043905877, 0.003138068, -0.049992483, 0.15067059, 0.13143604, -0.0041858954, 0.08278651, 0.03127779, 0.0070829904, 0.046700083, -0.04111167, 0.054324586, -0.08365293, -0.118353084, -0.082180016, -0.07091655, 0.03712613, -0.009102833, -0.05285167, -0.04704665, -0.11662024, -0.025234507, 0.016386185, 0.048302963, 0.0021890665, 0.00025231123, -0.0574437, 0.055407614, 0.05111883, 0.005734623, -0.011371774, 0.08581898, -0.056793887, -0.015530594, 0.07754467, -0.03158104, -0.016505318, 0.020512514, -0.061039347, -0.0407651, -0.044295765, -0.036909528, -0.12883678, 0.057920232, 0.018205667, 0.00092057214, 0.09210054, 0.11185493, -0.1258043, -0.028440263, -0.011685851, -0.06294547, -0.019841038, 0.050555658, -0.038382445, -0.028180338, 0.005041486, -0.06927034, 0.12285847, -0.05363145, 0.021682182, 0.14235294, 0.012530611, -0.014707494, 0.038923956, 0.039812036, 0.011512567, 0.09374674, -0.04791307, -0.033855394, -0.045010563, -0.055061042, 0.10310408, 0.027313916, 0.011512567, 0.030259747, -0.04204307, 0.07303928, 0.0815302, -0.008122696, -0.12745051, -0.09781891, -0.019981831, 0.02417314, -0.073472485, -0.034310266, -0.040895063, 0.014664173, -0.065501414, -0.05614407, -0.0007351039, 0.040895063, 0.034223624, 0.022180373, -0.011144338, 0.096779205, 0.012162383, 0.01907209, 0.16505317, -0.06502488, 0.044793956, 0.039205544, -0.007770712, 0.045573737, 0.111595005, -0.063508645], index=1), Embedding(embedding=[0.055870138, 0.09400151, 0.19684912, 0.14582106, 0.05312319, 0.05340254, 0.13427457, -0.091114886, 0.07300365, -0.011499937, -0.004443422, -0.04378822, -0.062015854, 0.077706054, 0.03089153, -0.007117623, -0.029052472, 0.013641626, 0.058663644, 0.0074609914, 0.05670819, 0.081477284, 0.036036238, 0.03240468, -0.03871335, -0.043439034, -0.00991113, 0.008962502, 0.16621366, 0.038503837, 0.009998427, -0.0318227, -0.019403234, -0.019542908, 0.04027306, 0.09786586, 0.048094876, -0.043625265, 0.004926466, -0.017634012, -0.006937209, 0.04229835, -0.0785441, -0.054473385, -0.015806593, 0.02791179, -0.031240718, -0.061131243, 0.14228262, 0.01291997, 0.08915943, -0.09185982, 0.057779036, -0.11378885, -0.02488549, -0.11853781, -0.021579841, 0.05675475, -0.09474644, -0.037619226, 0.13027054, 0.03901598, 0.03687429, 0.057732478, -0.03449981, 0.083293065, 0.017180068, -0.016085943, -0.11276456, 0.037968416, 0.02133541, -0.11602365, 0.093070336, -0.0129316095, 0.0098296525, 0.0150151, 0.16826223, -0.025560588, 0.0600604, -0.003363848, 0.05275072, 0.05912923, -0.12961872, 0.07607651, -0.027190134, 0.012058638, 0.02914559, -0.08636592, -0.081803195, 0.062528, -0.0001176692, -0.055078644, 0.017634012, 0.019880457, 0.0055724643, 0.00779854, 0.019857178, 0.01889109, 0.026515037, 0.07486598, 0.00330274, -0.08957846, -0.012512583, -0.0072805774, -0.014619353, -0.050515916, 0.0031368756, 0.060432866, 0.1008456, 0.0077228826, -0.0092593115, -0.012349629, 0.0043066563, 0.062248647, -0.055404555, -0.0052407356, -0.09032339, 0.043322638, 0.08739021, 0.002338107, 0.0068790107, 0.0043008365, 0.036152635, 0.08422423, 0.10382534, -0.09544482, 0.056615073, 0.027190134, -0.11751352, 0.018495344, -0.007897477, 0.044602994, 0.099076375, -0.07090852, -0.114813134, -0.09223229, 0.0013152761, -0.029005913, 0.09185982, 0.015550522, 0.06001384, 0.0708154, 0.052611046, 0.037107084, -0.018763054, -0.053961243, 0.08129105, -0.011953882, 0.027120296, 0.09656222, 0.15466717, -0.0026261874, 0.033079777, 0.09730716, 0.05763936, -0.09916949, 0.11323015, 0.03871335, -0.011651251, -0.101311184, -0.005013763, 0.0300302, -0.023558576, -0.08152384, 0.06690449, 0.04800176, 0.033405688, -0.0028677094, -0.06662514, -0.032218445, -0.050562475, 0.008153548, 0.09963508, 0.049026046, -0.04732666, 0.08245501, -0.08129105, 0.111460924, 0.0611778, 0.043066565, 0.05191267, 0.021160815, -0.027678998, -0.013478671, -0.04444004, -0.038643513, -0.033522084, -0.022545928, -0.02366333, 0.05079527, 0.027050458, 0.022254938, -0.03901598, -0.03985403, 0.07980118, 0.0138976965, 0.07756638, 0.12319365, -0.08585378, -0.03086825, -0.0300302, -0.021067698, -0.011331162, 0.04115767, -0.06806845, -0.021975588, -0.061084684, 0.0129316095, 0.09823833, -0.08375865, 0.035244744, 0.13203976, 0.052424815, 0.07221215, 0.13511261, 0.084457025, -0.063785076, 0.08939222, -0.04949163, 0.054566503, -0.019030767, 0.09102177, 0.029331822, 0.049957216, -0.04194916, -0.029774128, -0.039039258, 0.0073387753, 0.094932675, -0.0047751507, -0.018472064, 0.007466811, -0.012046998, -0.01617906, -0.06825469, -0.13352963, 0.06434377, 0.055404555, 0.010510569, 0.0057761576, 0.0708154, 0.00071692734, 0.019391594, -0.038434, -0.021859191, 0.035268024, -0.013746382, -0.0014505867, 0.05614949, -0.015620359, 0.00930005, 0.08492261, -0.036711335, 0.008787907, 0.013909336, -0.05577702], index=2), Embedding(embedding=[0.019417854, 0.10419106, 0.1256144, 0.04534765, 0.057522148, 0.0006315079, -0.015631018, -0.008452547, 0.12806818, 0.045701556, 0.07668047, 0.022874372, -0.029044202, -0.016150085, 0.02720387, 0.0270859, -0.086259626, -0.013401386, -0.039826654, 0.09281875, 0.035461765, 0.013672717, 0.024124853, 0.056342445, -0.0631847, 0.07724672, -0.056908704, 0.04032213, 0.06993259, -0.0037543937, 0.14156394, -0.034659572, -0.08347554, -0.03819867, 0.1507184, 0.05487962, -0.10702234, -0.12740754, 0.016692748, 0.024301808, 0.009526073, -0.102209166, -0.09862287, -0.09093124, -0.03602802, -0.039071646, 0.0072610504, -0.025292756, 0.06629911, 0.12901193, 0.092865944, -0.09546128, 0.095272526, -0.068611324, -0.0074674976, -0.0053056986, -0.012646378, 0.06648786, -0.09234688, -0.017105643, 0.11438366, 0.07861517, -0.021093026, -0.006388073, -0.04074682, 0.05351117, 0.061108433, -0.02249687, -0.07965331, 0.04034572, 0.025575884, -0.007974768, 0.072433546, -0.026944336, 0.099094756, -0.0036452713, 0.069177575, -0.023959696, -0.03815148, 0.06582723, -0.062193755, 0.02649605, -0.07587827, 0.047919393, -0.03921321, 0.07120666, -0.059268102, 0.023487816, -0.16317603, 0.064081274, -0.029634051, 0.022956952, 0.0037661905, 0.03560333, -0.045324054, 0.09149749, 0.034871917, 0.031096878, 0.06544973, 0.09956664, -0.0033179047, -0.06681818, 0.072008856, 0.011755705, 0.058277152, -0.03779757, -0.0028740428, 0.057097454, 0.041431047, -0.018509485, 0.0043766852, 0.0073554264, -0.031120472, 0.11740369, -0.005004875, -0.004261664, -0.11136363, 0.09239406, 0.005432516, -0.05695589, -0.021859832, 0.085551806, 0.107682966, 0.047423918, 0.08687307, -0.04501733, -0.032135013, 0.014356943, -0.08814714, -0.04360169, 0.037042562, 0.06327908, 0.06573285, -0.09451752, -0.06375096, -0.07545358, -0.04223324, 0.04973613, -0.01581977, 0.026944336, 0.07465138, 0.03555614, -0.03659428, 0.062052194, -0.076963596, -0.045748744, -0.024301808, -0.04695204, -0.011667227, 0.13986517, 0.070451654, -0.0092606405, -0.03003515, 0.12854005, 0.006736084, -0.030554216, 0.031615946, 0.13826078, -0.021482328, -0.07668047, -0.053322416, 0.050396763, -0.05322804, -0.09805662, 0.0120447315, -0.12825693, 0.009531972, -0.075123265, -0.07139541, -0.0278645, -0.018745424, -0.038953677, 0.057003077, 0.06077812, -0.023535004, 0.014675462, -0.08956278, 0.03234736, 0.121367484, 0.0069307345, -0.011584649, 0.19498073, -0.064836286, -0.057097454, -0.014026627, 0.04183214, -0.1252369, 0.012823333, 0.010121821, 0.025033223, -0.051482085, -0.010281081, 0.0108237425, 0.0011288249, 0.05936248, 0.04067604, 0.015690003, 0.0604478, 0.007019212, -0.08998748, -0.037915543, -0.045913905, 0.0020497278, 0.017825259, -0.11466679, -0.010239791, -0.03843461, -0.024136651, -0.0071253846, -0.011590547, -0.03624037, 0.083286785, -0.012375047, 0.05832434, 0.024868065, 0.11947996, -0.006199321, 0.045701556, 0.005618319, -0.07399075, -0.089468405, 0.0043353955, -0.035744894, 0.07488732, 0.048509244, 0.023133907, -0.041077137, -0.023582192, 0.10466294, -0.05558744, -0.04803736, -0.017494943, 0.031096878, 0.015454063, -0.0565312, -0.0777186, 0.06606317, 0.0318047, 0.040794007, -0.07601983, -0.024101261, 0.032418143, -0.044922955, -0.08578774, 0.044333108, 0.094423145, 0.0215885, -0.08876059, 0.07205604, -0.029067796, 0.07672765, -0.027722938, 0.07517045, 0.010959408, 0.09291313, -0.07375481], index=3)], model='text-embedding-3-small', usage=Usage(prompt_tokens=207, total_tokens=207), error=None, raw_response=None, input=['Renewable energy technologies not only help in reducing greenhouse gas emissions but also contribute significantly to the economy by creating jobs in the manufacturing and installation sectors. The growth in renewable energy usage boosts local economies through increased investment in technology and infrastructure.', 'Solar panels convert sunlight into electricity by allowing photons, or light particles, to knock electrons free from atoms, generating a flow of electricity. Solar panels are a type of renewable energy technology that has been found to have a significant positive effect on the environment by reducing the reliance on fossil fuels.', 'While solar energy offers substantial environmental benefits, such as reducing carbon footprints and pollution, it also has downsides. The production of solar panels can lead to hazardous waste, and large solar farms require significant land, which can disrupt local ecosystems.', 'Renewable energy sources like wind, solar, and hydro power play a crucial role in combating climate change. They do not produce greenhouse gases during operation, making them essential for sustainable development. However, the initial setup and material sourcing for these technologies can still have environmental impacts.'])\n"
     ]
    }
   ],
   "source": [
    "# the documents can fit into a batch, thus we only need the simple embedder\n",
    "# embedder takes a list of string. we will pass only the content of the documents\n",
    "output = embedder(input=[doc[\"content\"] for doc in documents])\n",
    "print(output.embedding_dim, output.length, output)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "FAISSRetriever(\n",
       "  top_k=2, metric=prob, dimensions=256, total_documents=4\n",
       "  (embedder): Embedder(\n",
       "    model_kwargs={'model': 'text-embedding-3-small', 'dimensions': 256, 'encoding_format': 'float'}, \n",
       "    (model_client): OpenAIClient()\n",
       "  )\n",
       ")"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# prepare the retriever\n",
    "\n",
    "from lightrag.components.retriever import FAISSRetriever\n",
    "\n",
    "# pass the documents in the initialization \n",
    "documents_embeddings = [x.embedding for x in output.data]\n",
    "retriever = FAISSRetriever(top_k=2, embedder=embedder, documents=documents_embeddings)\n",
    "retriever"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[RetrieverOutput(doc_indices=[0, 3], doc_scores=[0.8119999766349792, 0.7749999761581421], query='What are the benefits of renewable energy?', documents=None)]\n",
      "[RetrieverOutput(doc_indices=[2, 1], doc_scores=[0.8169999718666077, 0.8109999895095825], query='How do solar panels impact the environment?', documents=None)]\n",
      "[RetrieverOutput(doc_indices=[0, 3], doc_scores=[0.8119999766349792, 0.7749999761581421], query='What are the benefits of renewable energy?', documents=None), RetrieverOutput(doc_indices=[2, 1], doc_scores=[0.8169999718666077, 0.8109999895095825], query='How do solar panels impact the environment?', documents=None)]\n"
     ]
    }
   ],
   "source": [
    "# execute the retriever\n",
    "output_1 = retriever(input=query_1)\n",
    "output_2 = retriever(input=query_2)\n",
    "output_3 = retriever(input = [query_1, query_2])\n",
    "print(output_1)\n",
    "print(output_2)\n",
    "print(output_3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "FAISSRetriever(\n",
      "  top_k=2, metric=prob\n",
      "  (embedder): Embedder(\n",
      "    model_kwargs={'model': 'text-embedding-3-small', 'dimensions': 256, 'encoding_format': 'float'}, \n",
      "    (model_client): OpenAIClient()\n",
      "  )\n",
      ")\n",
      "FAISSRetriever(\n",
      "  top_k=2, metric=prob, dimensions=256, total_documents=4\n",
      "  (embedder): Embedder(\n",
      "    model_kwargs={'model': 'text-embedding-3-small', 'dimensions': 256, 'encoding_format': 'float'}, \n",
      "    (model_client): OpenAIClient()\n",
      "  )\n",
      ")\n",
      "[RetrieverOutput(doc_indices=[0, 3], doc_scores=[0.8119999766349792, 0.7749999761581421], query='What are the benefits of renewable energy?', documents=None)]\n",
      "[RetrieverOutput(doc_indices=[2, 1], doc_scores=[0.8169999718666077, 0.8109999895095825], query='How do solar panels impact the environment?', documents=None)]\n",
      "[RetrieverOutput(doc_indices=[0, 3], doc_scores=[0.8119999766349792, 0.7749999761581421], query='What are the benefits of renewable energy?', documents=None), RetrieverOutput(doc_indices=[2, 1], doc_scores=[0.8169999718666077, 0.8109999895095825], query='How do solar panels impact the environment?', documents=None)]\n"
     ]
    }
   ],
   "source": [
    "# second, we dont pass documents in init, and instead pass it with method build_index_from_documents\n",
    "\n",
    "retriever_1 = FAISSRetriever(top_k=2, embedder=embedder)\n",
    "print(retriever_1)\n",
    "retriever_1.build_index_from_documents(documents=documents_embeddings)\n",
    "print(retriever_1)\n",
    "\n",
    "output_1 = retriever_1(input=query_1)\n",
    "output_2 = retriever_1(input=query_2)\n",
    "output_3 = retriever_1(input = [query_1, query_2])\n",
    "print(output_1)\n",
    "print(output_2)\n",
    "print(output_3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## BM25 retriever (simple)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "InMemoryBM25Retriever(top_k=2, k1=1.5, b=0.75, epsilon=0.25, use_tokenizer=True, total_documents=4)\n"
     ]
    }
   ],
   "source": [
    "from lightrag.components.retriever import BM25Retriever\n",
    "\n",
    "document_map_func = lambda x: x[\"content\"]\n",
    "\n",
    "bm25_retriever = BM25Retriever(top_k=2, documents=documents, document_map_func=document_map_func)\n",
    "print(bm25_retriever)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['what', 'are', 'the', 'benefits', 'of', 'renewable', 'energy?']\n",
      "['what', 'are', 'the', 'benef', 'its', 'of', 're', 'new', 'able', 'energy', '?']\n"
     ]
    }
   ],
   "source": [
    "# show how a word splitter and a token splitter differs\n",
    "\n",
    "from lightrag.components.retriever.bm25_retriever import split_text_by_word_fn_then_lower_tokenized, split_text_by_word_fn\n",
    "\n",
    "query_1_words = split_text_by_word_fn(query_1)\n",
    "query_1_tokens = split_text_by_word_fn_then_lower_tokenized(query_1)\n",
    "\n",
    "print(query_1_words)\n",
    "print(query_1_tokens)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[RetrieverOutput(doc_indices=[2, 1], doc_scores=[2.151683837681807, 1.6294762236217233], query='What are the benefits of renewable energy?', documents=None)]\n",
      "[RetrieverOutput(doc_indices=[3, 2], doc_scores=[1.5166601493236314, 0.7790170272403408], query='How do solar panels impact the environment?', documents=None)]\n",
      "[RetrieverOutput(doc_indices=[2, 1], doc_scores=[2.151683837681807, 1.6294762236217233], query='What are the benefits of renewable energy?', documents=None), RetrieverOutput(doc_indices=[3, 2], doc_scores=[1.5166601493236314, 0.7790170272403408], query='How do solar panels impact the environment?', documents=None)]\n"
     ]
    }
   ],
   "source": [
    "output_1 = bm25_retriever(input=query_1)\n",
    "output_2 = bm25_retriever(input=query_2)\n",
    "output_3 = bm25_retriever(input = [query_1, query_2])\n",
    "print(output_1)\n",
    "print(output_2)\n",
    "print(output_3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'type': 'InMemoryBM25Retriever', 'data': {'_components': {'_ordered_dict': True, 'data': []}, '_parameters': {'_ordered_dict': True, 'data': []}, 'training': False, '_init_args': {'b': 0.75, 'document_map_func': None, 'documents': None, 'epsilon': 0.25, 'k1': 1.5, 'top_k': 5, 'use_tokenizer': True}, 'k1': 1.5, 'b': 0.75, 'epsilon': 0.25, 'top_k': 2, '_use_tokenizer': True, '_split_function': <function split_text_by_word_fn_then_lower_tokenized at 0x10f919f80>, 'indexed': True, 'index_keys': ['avgdl', 'b', 'doc_len', 'epsilon', 'idf', 'indexed', 'k1', 'nd', 't2d', 'top_k', 'total_documents', 'use_tokenizer'], 't2d': [{'.': 2, 'able': 2, 'also': 1, 'and': 2, 'ased': 1, 'boost': 1, 'but': 1, 'by': 1, 'cing': 1, 'con': 1, 'conom': 1, 'conomy': 1, 'creating': 1, 'du': 1, 'e': 2, 'em': 1, 'energy': 2, 'gas': 1, 'green': 1, 'growth': 1, 'help': 1, 'house': 1, 'ies': 1, 'ificantly': 1, 'in': 4, 'incre': 1, 'inf': 1, 'installation': 1, 'investment': 1, 'issions': 1, 'jobs': 1, 'local': 1, 'manufact': 1, 'new': 2, 'not': 1, 'ologies': 1, 'only': 1, 'ors': 1, 'rastructure': 1, 're': 3, 's': 1, 'sect': 1, 'sign': 1, 'techn': 1, 'technology': 1, 'the': 3, 'through': 1, 'to': 1, 'tribute': 1, 'uring': 1, 'usage': 1}, {',': 3, '.': 2, 'a': 3, 'able': 1, 'allow': 1, 'ance': 1, 'are': 1, 'ating': 1, 'atoms': 1, 'been': 1, 'by': 2, 'cing': 1, 'convert': 1, 'du': 1, 'effect': 1, 'elect': 1, 'electric': 2, 'els': 1, 'energy': 1, 'environment': 1, 'f': 1, 'flow': 1, 'found': 1, 'free': 1, 'from': 1, 'fu': 1, 'gener': 1, 'has': 1, 'have': 1, 'ing': 1, 'into': 1, 'ity': 2, 'kn': 1, 'li': 1, 'light': 2, 'new': 1, 'ock': 1, 'of': 2, 'olar': 2, 'on': 2, 'ons': 1, 'or': 1, 'ossil': 1, 'panels': 2, 'particles': 1, 'phot': 1, 'positive': 1, 're': 3, 'rons': 1, 's': 2, 'significant': 1, 'sun': 1, 'technology': 1, 'that': 1, 'the': 2, 'to': 2, 'type': 1}, {',': 4, '.': 2, 'al': 1, 'also': 1, 'and': 2, 'arms': 1, 'as': 1, 'aste': 1, 'azard': 1, 'benef': 1, 'can': 2, 'carbon': 1, 'cing': 1, 'dis': 1, 'down': 1, 'du': 1, 'ec': 1, 'energy': 1, 'environment': 1, 'f': 1, 'foot': 1, 'h': 1, 'has': 1, 'ides': 1, 'it': 1, 'its': 1, 'land': 1, 'large': 1, 'lead': 1, 'local': 1, 'of': 1, 'offers': 1, 'olar': 3, 'os': 1, 'ous': 1, 'panels': 1, 'poll': 1, 'prints': 1, 'production': 1, 're': 1, 'require': 1, 'rupt': 1, 's': 4, 'significant': 1, 'stantial': 1, 'sub': 1, 'such': 1, 'the': 1, 'to': 1, 'ution': 1, 'w': 1, 'which': 1, 'while': 1, 'ystems': 1}, {',': 4, '.': 3, 'a': 1, 'able': 1, 'acts': 1, 'al': 1, 'and': 2, 'ases': 1, 'ating': 1, 'can': 1, 'change': 1, 'climate': 1, 'comb': 1, 'cr': 1, 'development': 1, 'do': 1, 'during': 1, 'energy': 1, 'environment': 1, 'essential': 1, 'for': 2, 'g': 1, 'green': 1, 'have': 1, 'house': 1, 'however': 1, 'hydro': 1, 'ial': 1, 'imp': 1, 'in': 1, 'initial': 1, 'like': 1, 'making': 1, 'material': 1, 'new': 1, 'not': 1, 'olar': 1, 'ologies': 1, 'operation': 1, 'ourcing': 1, 'play': 1, 'power': 1, 'produce': 1, 're': 1, 'role': 1, 's': 3, 'setup': 1, 'sources': 1, 'still': 1, 'techn': 1, 'the': 1, 'them': 1, 'these': 1, 'they': 1, 'uc': 1, 'ustainable': 1, 'wind': 1}], 'nd': {',': 3, '.': 4, 'a': 2, 'able': 3, 'acts': 1, 'al': 2, 'allow': 1, 'also': 2, 'ance': 1, 'and': 3, 'are': 1, 'arms': 1, 'as': 1, 'ased': 1, 'ases': 1, 'aste': 1, 'ating': 2, 'atoms': 1, 'azard': 1, 'been': 1, 'benef': 1, 'boost': 1, 'but': 1, 'by': 2, 'can': 2, 'carbon': 1, 'change': 1, 'cing': 3, 'climate': 1, 'comb': 1, 'con': 1, 'conom': 1, 'conomy': 1, 'convert': 1, 'cr': 1, 'creating': 1, 'development': 1, 'dis': 1, 'do': 1, 'down': 1, 'du': 3, 'during': 1, 'e': 1, 'ec': 1, 'effect': 1, 'elect': 1, 'electric': 1, 'els': 1, 'em': 1, 'energy': 4, 'environment': 3, 'essential': 1, 'f': 2, 'flow': 1, 'foot': 1, 'for': 1, 'found': 1, 'free': 1, 'from': 1, 'fu': 1, 'g': 1, 'gas': 1, 'gener': 1, 'green': 2, 'growth': 1, 'h': 1, 'has': 2, 'have': 2, 'help': 1, 'house': 2, 'however': 1, 'hydro': 1, 'ial': 1, 'ides': 1, 'ies': 1, 'ificantly': 1, 'imp': 1, 'in': 2, 'incre': 1, 'inf': 1, 'ing': 1, 'initial': 1, 'installation': 1, 'into': 1, 'investment': 1, 'issions': 1, 'it': 1, 'its': 1, 'ity': 1, 'jobs': 1, 'kn': 1, 'land': 1, 'large': 1, 'lead': 1, 'li': 1, 'light': 1, 'like': 1, 'local': 2, 'making': 1, 'manufact': 1, 'material': 1, 'new': 3, 'not': 2, 'ock': 1, 'of': 2, 'offers': 1, 'olar': 3, 'ologies': 2, 'on': 1, 'only': 1, 'ons': 1, 'operation': 1, 'or': 1, 'ors': 1, 'os': 1, 'ossil': 1, 'ourcing': 1, 'ous': 1, 'panels': 2, 'particles': 1, 'phot': 1, 'play': 1, 'poll': 1, 'positive': 1, 'power': 1, 'prints': 1, 'produce': 1, 'production': 1, 'rastructure': 1, 're': 4, 'require': 1, 'role': 1, 'rons': 1, 'rupt': 1, 's': 4, 'sect': 1, 'setup': 1, 'sign': 1, 'significant': 2, 'sources': 1, 'stantial': 1, 'still': 1, 'sub': 1, 'such': 1, 'sun': 1, 'techn': 2, 'technology': 2, 'that': 1, 'the': 4, 'them': 1, 'these': 1, 'they': 1, 'through': 1, 'to': 3, 'tribute': 1, 'type': 1, 'uc': 1, 'uring': 1, 'usage': 1, 'ustainable': 1, 'ution': 1, 'w': 1, 'which': 1, 'while': 1, 'wind': 1, 'ystems': 1}, 'idf': {',': 0.1404089065375674, '.': 0.1404089065375674, 'a': 0.0, 'able': 0.1404089065375674, 'acts': 0.8472978603872037, 'al': 0.0, 'allow': 0.8472978603872037, 'also': 0.0, 'ance': 0.8472978603872037, 'and': 0.1404089065375674, 'are': 0.8472978603872037, 'arms': 0.8472978603872037, 'as': 0.8472978603872037, 'ased': 0.8472978603872037, 'ases': 0.8472978603872037, 'aste': 0.8472978603872037, 'ating': 0.0, 'atoms': 0.8472978603872037, 'azard': 0.8472978603872037, 'been': 0.8472978603872037, 'benef': 0.8472978603872037, 'boost': 0.8472978603872037, 'but': 0.8472978603872037, 'by': 0.0, 'can': 0.0, 'carbon': 0.8472978603872037, 'change': 0.8472978603872037, 'cing': 0.1404089065375674, 'climate': 0.8472978603872037, 'comb': 0.8472978603872037, 'con': 0.8472978603872037, 'conom': 0.8472978603872037, 'conomy': 0.8472978603872037, 'convert': 0.8472978603872037, 'cr': 0.8472978603872037, 'creating': 0.8472978603872037, 'development': 0.8472978603872037, 'dis': 0.8472978603872037, 'do': 0.8472978603872037, 'down': 0.8472978603872037, 'du': 0.1404089065375674, 'during': 0.8472978603872037, 'e': 0.8472978603872037, 'ec': 0.8472978603872037, 'effect': 0.8472978603872037, 'elect': 0.8472978603872037, 'electric': 0.8472978603872037, 'els': 0.8472978603872037, 'em': 0.8472978603872037, 'energy': 0.1404089065375674, 'environment': 0.1404089065375674, 'essential': 0.8472978603872037, 'f': 0.0, 'flow': 0.8472978603872037, 'foot': 0.8472978603872037, 'for': 0.8472978603872037, 'found': 0.8472978603872037, 'free': 0.8472978603872037, 'from': 0.8472978603872037, 'fu': 0.8472978603872037, 'g': 0.8472978603872037, 'gas': 0.8472978603872037, 'gener': 0.8472978603872037, 'green': 0.0, 'growth': 0.8472978603872037, 'h': 0.8472978603872037, 'has': 0.0, 'have': 0.0, 'help': 0.8472978603872037, 'house': 0.0, 'however': 0.8472978603872037, 'hydro': 0.8472978603872037, 'ial': 0.8472978603872037, 'ides': 0.8472978603872037, 'ies': 0.8472978603872037, 'ificantly': 0.8472978603872037, 'imp': 0.8472978603872037, 'in': 0.0, 'incre': 0.8472978603872037, 'inf': 0.8472978603872037, 'ing': 0.8472978603872037, 'initial': 0.8472978603872037, 'installation': 0.8472978603872037, 'into': 0.8472978603872037, 'investment': 0.8472978603872037, 'issions': 0.8472978603872037, 'it': 0.8472978603872037, 'its': 0.8472978603872037, 'ity': 0.8472978603872037, 'jobs': 0.8472978603872037, 'kn': 0.8472978603872037, 'land': 0.8472978603872037, 'large': 0.8472978603872037, 'lead': 0.8472978603872037, 'li': 0.8472978603872037, 'light': 0.8472978603872037, 'like': 0.8472978603872037, 'local': 0.0, 'making': 0.8472978603872037, 'manufact': 0.8472978603872037, 'material': 0.8472978603872037, 'new': 0.1404089065375674, 'not': 0.0, 'ock': 0.8472978603872037, 'of': 0.0, 'offers': 0.8472978603872037, 'olar': 0.1404089065375674, 'ologies': 0.0, 'on': 0.8472978603872037, 'only': 0.8472978603872037, 'ons': 0.8472978603872037, 'operation': 0.8472978603872037, 'or': 0.8472978603872037, 'ors': 0.8472978603872037, 'os': 0.8472978603872037, 'ossil': 0.8472978603872037, 'ourcing': 0.8472978603872037, 'ous': 0.8472978603872037, 'panels': 0.0, 'particles': 0.8472978603872037, 'phot': 0.8472978603872037, 'play': 0.8472978603872037, 'poll': 0.8472978603872037, 'positive': 0.8472978603872037, 'power': 0.8472978603872037, 'prints': 0.8472978603872037, 'produce': 0.8472978603872037, 'production': 0.8472978603872037, 'rastructure': 0.8472978603872037, 're': 0.1404089065375674, 'require': 0.8472978603872037, 'role': 0.8472978603872037, 'rons': 0.8472978603872037, 'rupt': 0.8472978603872037, 's': 0.1404089065375674, 'sect': 0.8472978603872037, 'setup': 0.8472978603872037, 'sign': 0.8472978603872037, 'significant': 0.0, 'sources': 0.8472978603872037, 'stantial': 0.8472978603872037, 'still': 0.8472978603872037, 'sub': 0.8472978603872037, 'such': 0.8472978603872037, 'sun': 0.8472978603872037, 'techn': 0.0, 'technology': 0.0, 'that': 0.8472978603872037, 'the': 0.1404089065375674, 'them': 0.8472978603872037, 'these': 0.8472978603872037, 'they': 0.8472978603872037, 'through': 0.8472978603872037, 'to': 0.1404089065375674, 'tribute': 0.8472978603872037, 'type': 0.8472978603872037, 'uc': 0.8472978603872037, 'uring': 0.8472978603872037, 'usage': 0.8472978603872037, 'ustainable': 0.8472978603872037, 'ution': 0.8472978603872037, 'w': 0.8472978603872037, 'which': 0.8472978603872037, 'while': 0.8472978603872037, 'wind': 0.8472978603872037, 'ystems': 0.8472978603872037}, 'doc_len': [64, 65, 66, 75], 'avgdl': 67.5, 'total_documents': 4, 'documents': [{'content': 'Renewable energy technologies not only help in reducing greenhouse gas emissions but also contribute significantly to the economy by creating jobs in the manufacturing and installation sectors. The growth in renewable energy usage boosts local economies through increased investment in technology and infrastructure.', 'title': 'The Impact of Renewable Energy on the Economy'}, {'content': 'Solar panels convert sunlight into electricity by allowing photons, or light particles, to knock electrons free from atoms, generating a flow of electricity. Solar panels are a type of renewable energy technology that has been found to have a significant positive effect on the environment by reducing the reliance on fossil fuels.', 'title': 'Understanding Solar Panels'}, {'content': 'While solar energy offers substantial environmental benefits, such as reducing carbon footprints and pollution, it also has downsides. The production of solar panels can lead to hazardous waste, and large solar farms require significant land, which can disrupt local ecosystems.', 'title': 'Pros and Cons of Solar Energy'}, {'content': 'Renewable energy sources like wind, solar, and hydro power play a crucial role in combating climate change. They do not produce greenhouse gases during operation, making them essential for sustainable development. However, the initial setup and material sourcing for these technologies can still have environmental impacts.', 'title': 'Renewable Energy and Its Effects'}], 'tokenized_documents': [[',', ',', ',', ',', '.', '.', '.', 'a', 'able', 'acts', 'al', 'and', 'and', 'ases', 'ating', 'can', 'change', 'climate', 'comb', 'cr', 'development', 'do', 'during', 'energy', 'environment', 'essential', 'for', 'for', 'g', 'green', 'have', 'house', 'however', 'hydro', 'ial', 'imp', 'in', 'initial', 'like', 'making', 'material', 'new', 'not', 'olar', 'ologies', 'operation', 'ourcing', 'play', 'power', 'produce', 're', 'role', 's', 's', 's', 'setup', 'sources', 'still', 'techn', 'the', 'them', 'these', 'they', 'uc', 'ustainable', 'wind'], [',', ',', ',', ',', '.', '.', 'al', 'also', 'and', 'and', 'arms', 'as', 'aste', 'azard', 'benef', 'can', 'can', 'carbon', 'cing', 'dis', 'down', 'du', 'ec', 'energy', 'environment', 'f', 'foot', 'h', 'has', 'ides', 'it', 'its', 'land', 'large', 'lead', 'local', 'of', 'offers', 'olar', 'olar', 'olar', 'os', 'ous', 'panels', 'poll', 'prints', 'production', 're', 'require', 'rupt', 's', 's', 's', 's', 'significant', 'stantial', 'sub', 'such', 'the', 'to', 'ution', 'w', 'which', 'while', 'ystems'], [',', ',', ',', '.', '.', 'a', 'a', 'a', 'able', 'allow', 'ance', 'are', 'ating', 'atoms', 'been', 'by', 'by', 'cing', 'convert', 'du', 'effect', 'elect', 'electric', 'electric', 'els', 'energy', 'environment', 'f', 'flow', 'found', 'free', 'from', 'fu', 'gener', 'has', 'have', 'ing', 'into', 'ity', 'ity', 'kn', 'li', 'light', 'light', 'new', 'ock', 'of', 'of', 'olar', 'olar', 'on', 'on', 'ons', 'or', 'ossil', 'panels', 'panels', 'particles', 'phot', 'positive', 're', 're', 're', 'rons', 's', 's', 'significant', 'sun', 'technology', 'that', 'the', 'the', 'to', 'to', 'type'], ['.', '.', 'able', 'able', 'also', 'and', 'and', 'ased', 'boost', 'but', 'by', 'cing', 'con', 'conom', 'conomy', 'creating', 'du', 'e', 'e', 'em', 'energy', 'energy', 'gas', 'green', 'growth', 'help', 'house', 'ies', 'ificantly', 'in', 'in', 'in', 'in', 'incre', 'inf', 'installation', 'investment', 'issions', 'jobs', 'local', 'manufact', 'new', 'new', 'not', 'ologies', 'only', 'ors', 'rastructure', 're', 're', 're', 's', 'sect', 'sign', 'techn', 'technology', 'the', 'the', 'the', 'through', 'to', 'tribute', 'uring', 'usage']], 'average_idf': 0.5616356261502696}}\n"
     ]
    }
   ],
   "source": [
    "states = bm25_retriever.to_dict()\n",
    "print(states)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[RetrieverOutput(doc_indices=[0, 1], doc_scores=[0.8490398606823998, 0.6288584231026185], query='renewable energy?', documents=None)]\n",
      "[RetrieverOutput(doc_indices=[2, 1], doc_scores=[0.49343959021478, 0.38733491458639285], query='solar panels?', documents=None)]\n",
      "[RetrieverOutput(doc_indices=[0, 1], doc_scores=[0.8490398606823998, 0.6288584231026185], query='renewable energy?', documents=None), RetrieverOutput(doc_indices=[2, 1], doc_scores=[0.49343959021478, 0.38733491458639285], query='solar panels?', documents=None)]\n"
     ]
    }
   ],
   "source": [
    "# use short queries, it performs slightly better\n",
    "\n",
    "query_1_short = \"renewable energy?\"  # gt is [0, 3]\n",
    "query_2_short = \"solar panels?\"  # gt is [1, 2]\n",
    "\n",
    "output_1 = bm25_retriever(input=query_1_short)\n",
    "output_2 = bm25_retriever(input=query_2_short)\n",
    "output_3 = bm25_retriever(input = [query_1_short, query_2_short])\n",
    "print(output_1)\n",
    "print(output_2)\n",
    "print(output_3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[{'title': 'The Impact of Renewable Energy on the Economy', 'content': 'Renewable energy technologies not only help in reducing greenhouse gas emissions but also contribute significantly to the economy by creating jobs in the manufacturing and installation sectors. The growth in renewable energy usage boosts local economies through increased investment in technology and infrastructure.'}, {'title': 'Understanding Solar Panels', 'content': 'Solar panels convert sunlight into electricity by allowing photons, or light particles, to knock electrons free from atoms, generating a flow of electricity. Solar panels are a type of renewable energy technology that has been found to have a significant positive effect on the environment by reducing the reliance on fossil fuels.'}, {'title': 'Pros and Cons of Solar Energy', 'content': 'While solar energy offers substantial environmental benefits, such as reducing carbon footprints and pollution, it also has downsides. The production of solar panels can lead to hazardous waste, and large solar farms require significant land, which can disrupt local ecosystems.'}, {'title': 'Renewable Energy and Its Effects', 'content': 'Renewable energy sources like wind, solar, and hydro power play a crucial role in combating climate change. They do not produce greenhouse gases during operation, making them essential for sustainable development. However, the initial setup and material sourcing for these technologies can still have environmental impacts.'}]\n",
      "[RetrieverOutput(doc_indices=[0, 3], doc_scores=[0.9498793313012154, 0.8031794089550072], query='renewable energy?', documents=None)]\n",
      "[RetrieverOutput(doc_indices=[2, 1], doc_scores=[0.5343238380789569, 0.4568096570283078], query='solar panels?', documents=None)]\n",
      "[RetrieverOutput(doc_indices=[0, 3], doc_scores=[0.9498793313012154, 0.8031794089550072], query='renewable energy?', documents=None), RetrieverOutput(doc_indices=[2, 1], doc_scores=[0.5343238380789569, 0.4568096570283078], query='solar panels?', documents=None)]\n"
     ]
    }
   ],
   "source": [
    "# use both title and content\n",
    "document_map_func = lambda x: x[\"title\"] + \" \" + x[\"content\"]\n",
    "\n",
    "print(documents)\n",
    "bm25_retriever.build_index_from_documents(documents=documents, document_map_func=document_map_func)\n",
    "\n",
    "output_1 = bm25_retriever(input=query_1_short)\n",
    "output_2 = bm25_retriever(input=query_2_short)\n",
    "output_3 = bm25_retriever(input = [query_1_short, query_2_short])\n",
    "print(output_1)\n",
    "print(output_2)\n",
    "print(output_3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Reranker (simple)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "# !poetry add cohere --group dev"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RerankerRetriever(\n",
      "  top_k=2, model_kwargs={'model': 'rerank-english-v3.0'}, model_client=CohereAPIClient(), total_documents=0\n",
      "  (model_client): CohereAPIClient()\n",
      ")\n"
     ]
    }
   ],
   "source": [
    "from lightrag.components.retriever import RerankerRetriever\n",
    "\n",
    "model_client = ModelClientType.COHERE()\n",
    "model_kwargs = {\"model\": \"rerank-english-v3.0\"}\n",
    "\n",
    "\n",
    "reranker = RerankerRetriever(\n",
    "    top_k=2, model_client=model_client, model_kwargs=model_kwargs\n",
    ")\n",
    "print(reranker)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RerankerRetriever(\n",
      "  top_k=2, model_kwargs={'model': 'rerank-english-v3.0', 'documents': ['Renewable energy technologies not only help in reducing greenhouse gas emissions but also contribute significantly to the economy by creating jobs in the manufacturing and installation sectors. The growth in renewable energy usage boosts local economies through increased investment in technology and infrastructure.', 'Solar panels convert sunlight into electricity by allowing photons, or light particles, to knock electrons free from atoms, generating a flow of electricity. Solar panels are a type of renewable energy technology that has been found to have a significant positive effect on the environment by reducing the reliance on fossil fuels.', 'While solar energy offers substantial environmental benefits, such as reducing carbon footprints and pollution, it also has downsides. The production of solar panels can lead to hazardous waste, and large solar farms require significant land, which can disrupt local ecosystems.', 'Renewable energy sources like wind, solar, and hydro power play a crucial role in combating climate change. They do not produce greenhouse gases during operation, making them essential for sustainable development. However, the initial setup and material sourcing for these technologies can still have environmental impacts.']}, model_client=CohereAPIClient(), total_documents=4\n",
      "  (model_client): CohereAPIClient()\n",
      ")\n"
     ]
    }
   ],
   "source": [
    "# build index and run queries\n",
    "document_map_func = lambda x: x[\"content\"]\n",
    "reranker.build_index_from_documents(documents=documents, document_map_func=document_map_func)\n",
    "\n",
    "print(reranker)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[RetrieverOutput(doc_indices=[0, 3], doc_scores=[0.99520767, 0.9696708], query='What are the benefits of renewable energy?', documents=None)]\n",
      "[RetrieverOutput(doc_indices=[1, 2], doc_scores=[0.98742366, 0.9701269], query='How do solar panels impact the environment?', documents=None)]\n",
      "[RetrieverOutput(doc_indices=[0, 3], doc_scores=[0.99520767, 0.9696708], query='What are the benefits of renewable energy?', documents=None), RetrieverOutput(doc_indices=[1, 2], doc_scores=[0.98742366, 0.9701269], query='How do solar panels impact the environment?', documents=None)]\n"
     ]
    }
   ],
   "source": [
    "# run queries\n",
    "output_1 = reranker(input=query_1)\n",
    "output_2 = reranker(input=query_2)\n",
    "output_3 = reranker(input = [query_1, query_2])\n",
    "print(output_1)\n",
    "print(output_2)\n",
    "print(output_3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RerankerRetriever(\n",
      "  top_k=2, model_kwargs={'model': 'BAAI/bge-reranker-base', 'documents': ['Renewable energy technologies not only help in reducing greenhouse gas emissions but also contribute significantly to the economy by creating jobs in the manufacturing and installation sectors. The growth in renewable energy usage boosts local economies through increased investment in technology and infrastructure.', 'Solar panels convert sunlight into electricity by allowing photons, or light particles, to knock electrons free from atoms, generating a flow of electricity. Solar panels are a type of renewable energy technology that has been found to have a significant positive effect on the environment by reducing the reliance on fossil fuels.', 'While solar energy offers substantial environmental benefits, such as reducing carbon footprints and pollution, it also has downsides. The production of solar panels can lead to hazardous waste, and large solar farms require significant land, which can disrupt local ecosystems.', 'Renewable energy sources like wind, solar, and hydro power play a crucial role in combating climate change. They do not produce greenhouse gases during operation, making them essential for sustainable development. However, the initial setup and material sourcing for these technologies can still have environmental impacts.']}, model_client=TransformersClient(), total_documents=4\n",
      "  (model_client): TransformersClient()\n",
      ")\n"
     ]
    }
   ],
   "source": [
    "# use transformer client\n",
    "\n",
    "model_client = ModelClientType.TRANSFORMERS()\n",
    "model_kwargs = {\"model\": \"BAAI/bge-reranker-base\"}\n",
    "\n",
    "reranker = RerankerRetriever(\n",
    "    top_k=2,\n",
    "    model_client=model_client,\n",
    "    model_kwargs=model_kwargs,\n",
    "    documents=documents,\n",
    "    document_map_func=document_map_func,\n",
    ")\n",
    "print(reranker)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "# run queries\n",
    "import torch\n",
    "# Set the number of threads for PyTorch, avoid segementation fault\n",
    "torch.set_num_threads(1)\n",
    "torch.set_num_interop_threads(1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[RetrieverOutput(doc_indices=[0, 3], doc_scores=[0.9996004700660706, 0.9950029253959656], query='What are the benefits of renewable energy?', documents=None)]\n",
      "[RetrieverOutput(doc_indices=[2, 0], doc_scores=[0.9994490742683411, 0.9994476437568665], query='How do solar panels impact the environment?', documents=None)]\n",
      "[RetrieverOutput(doc_indices=[0, 3], doc_scores=[0.9996004700660706, 0.9950029253959656], query='What are the benefits of renewable energy?', documents=None), RetrieverOutput(doc_indices=[2, 0], doc_scores=[0.9994490742683411, 0.9994476437568665], query='How do solar panels impact the environment?', documents=None)]\n"
     ]
    }
   ],
   "source": [
    "\n",
    "\n",
    "output_1 = reranker(input=query_1)\n",
    "output_2 = reranker(input=query_2)\n",
    "output_3 = reranker(input = [query_1, query_2])\n",
    "print(output_1)\n",
    "print(output_2)\n",
    "print(output_3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we see the second query is missing one. But Semantically, these documents might be close.\n",
    "If we use top_k = 3, the genearator might be able to filter out the irrelevant one and eventually give out the right final response."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[RetrieverOutput(doc_indices=[0, 3], doc_scores=[0.9844216108322144, 0.8057923913002014], query='What are the benefits of renewable energy?', documents=None)]\n",
      "[RetrieverOutput(doc_indices=[1, 2], doc_scores=[0.9824342131614685, 0.9368231892585754], query='How do solar panels impact the environment?', documents=None)]\n",
      "[RetrieverOutput(doc_indices=[0, 3], doc_scores=[0.9844216108322144, 0.8057923913002014], query='What are the benefits of renewable energy?', documents=None), RetrieverOutput(doc_indices=[1, 2], doc_scores=[0.9824342131614685, 0.9368231892585754], query='How do solar panels impact the environment?', documents=None)]\n"
     ]
    }
   ],
   "source": [
    "# try to use title this time\n",
    "document_map_func = lambda x: x[\"title\"] + \" \" + x[\"content\"]\n",
    "\n",
    "reranker.build_index_from_documents(documents=documents, document_map_func=document_map_func)\n",
    "\n",
    "# run queries\n",
    "output_1 = reranker(input=query_1)\n",
    "output_2 = reranker(input=query_2)\n",
    "output_3 = reranker(input = [query_1, query_2])\n",
    "print(output_1)\n",
    "print(output_2)\n",
    "print(output_3)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## LLM as retriever\n",
    "\n",
    "(1) Directly return the doc_indices from the LLM model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "LLMRetriever(\n",
      "  top_k=2, total_documents=4,\n",
      "  (generator): Generator(\n",
      "    model_kwargs={'model': 'gpt-4o'}, \n",
      "    (prompt): Prompt(\n",
      "      template: <SYS>\n",
      "      You are a retriever. Given a list of documents, you will retrieve the top_k {{top_k}} most relevant documents and output the indices (int) as a list:\n",
      "      [<index of the most relevant top_k options>]\n",
      "      <Documents>\n",
      "      {% for doc in documents %}\n",
      "      ```Index {{ loop.index - 1 }}. {{ doc }}```\n",
      "      ______________\n",
      "      {% endfor %}\n",
      "      </Documents>\n",
      "      </SYS>\n",
      "      Query: {{ input_str }}\n",
      "      You:\n",
      "      , preset_prompt_kwargs: {'top_k': 2, 'documents': ['Renewable energy technologies not only help in reducing greenhouse gas emissions but also contribute significantly to the economy by creating jobs in the manufacturing and installation sectors. The growth in renewable energy usage boosts local economies through increased investment in technology and infrastructure.', 'Solar panels convert sunlight into electricity by allowing photons, or light particles, to knock electrons free from atoms, generating a flow of electricity. Solar panels are a type of renewable energy technology that has been found to have a significant positive effect on the environment by reducing the reliance on fossil fuels.', 'While solar energy offers substantial environmental benefits, such as reducing carbon footprints and pollution, it also has downsides. The production of solar panels can lead to hazardous waste, and large solar farms require significant land, which can disrupt local ecosystems.', 'Renewable energy sources like wind, solar, and hydro power play a crucial role in combating climate change. They do not produce greenhouse gases during operation, making them essential for sustainable development. However, the initial setup and material sourcing for these technologies can still have environmental impacts.']}, prompt_variables: ['input_str', 'top_k', 'documents']\n",
      "    )\n",
      "    (model_client): OpenAIClient()\n",
      "    (output_processors): ListParser()\n",
      "  )\n",
      ")\n"
     ]
    }
   ],
   "source": [
    "from lightrag.components.retriever import LLMRetriever\n",
    "\n",
    "model_client = ModelClientType.OPENAI()\n",
    "model_kwargs = {\n",
    "    \"model\": \"gpt-4o\",\n",
    "}\n",
    "document_map_func = lambda x: x[\"content\"]\n",
    "llm_retriever = LLMRetriever(\n",
    "        top_k=2, \n",
    "        model_client=model_client, \n",
    "        model_kwargs=model_kwargs, \n",
    "        documents=documents, \n",
    "        document_map_func=document_map_func\n",
    "    )\n",
    "print(llm_retriever)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[RetrieverOutput(doc_indices=[0, 3], doc_scores=None, query='What are the benefits of renewable energy?', documents=None)]\n",
      "[RetrieverOutput(doc_indices=[1, 2], doc_scores=None, query='How do solar panels impact the environment?', documents=None)]\n",
      "[RetrieverOutput(doc_indices=[0, 3], doc_scores=None, query='What are the benefits of renewable energy?', documents=None), RetrieverOutput(doc_indices=[1, 2], doc_scores=None, query='How do solar panels impact the environment?', documents=None)]\n"
     ]
    }
   ],
   "source": [
    "# run queries\n",
    "output_1 = llm_retriever(input=query_1)\n",
    "output_2 = llm_retriever(input=query_2)\n",
    "output_3 = llm_retriever(input = [query_1, query_2])\n",
    "print(output_1)\n",
    "print(output_2)\n",
    "print(output_3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[RetrieverOutput(doc_indices=[0, 1], doc_scores=None, query='What are the benefits of renewable energy?', documents=None)]\n",
      "[RetrieverOutput(doc_indices=[1, 2], doc_scores=None, query='How do solar panels impact the environment?', documents=None)]\n",
      "[RetrieverOutput(doc_indices=[0, 1], doc_scores=None, query='What are the benefits of renewable energy?', documents=None), RetrieverOutput(doc_indices=[1, 2], doc_scores=None, query='How do solar panels impact the environment?', documents=None)]\n"
     ]
    }
   ],
   "source": [
    "# you should try both gpt-3.5-turbo and gpt-4o\n",
    "# you can use a different model without reinitializing the retriever\n",
    "model_kwargs = {\n",
    "    \"model\": \"gpt-3.5-turbo\",\n",
    "}\n",
    "output_1 = llm_retriever(model_kwargs=model_kwargs, input=query_1)\n",
    "output_2 = llm_retriever(model_kwargs=model_kwargs, input=query_2)\n",
    "output_3 = llm_retriever(model_kwargs=model_kwargs, input = [query_1, query_2])\n",
    "print(output_1)\n",
    "print(output_2)\n",
    "print(output_3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## LLMRetriever\n",
    "\n",
    "The indexing process is to form prompt using the targeting documents and set up the top_k parameter.\n",
    "The ``retrieve`` is to run the ``generator`` and parse the response to standard ``RetrieverOutputType`` which is a list of \n",
    "``RetrieverOutput``. Each ``RetrieverOutput`` contains the document id and the score."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--2024-06-19 19:53:19--  https://raw.githubusercontent.com/run-llama/llama_index/main/docs/docs/examples/data/paul_graham/paul_graham_essay.txt\n",
      "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.110.133, 185.199.111.133, 185.199.108.133, ...\n",
      "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.110.133|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 75042 (73K) [text/plain]\n",
      "Saving to: ‘data/paul_graham/paul_graham_essay.txt’\n",
      "\n",
      "data/paul_graham/pa 100%[===================>]  73.28K  --.-KB/s    in 0.02s   \n",
      "\n",
      "2024-06-19 19:53:19 (4.51 MB/s) - ‘data/paul_graham/paul_graham_essay.txt’ saved [75042/75042]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# prepare the document\n",
    "!mkdir -p 'data/paul_graham/'\n",
    "!wget 'https://raw.githubusercontent.com/run-llama/llama_index/main/docs/docs/examples/data/paul_graham/paul_graham_essay.txt' -O 'data/paul_graham/paul_graham_essay.txt'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: fsspec in /Users/liyin/Documents/test/LightRAG/.venv/lib/python3.11/site-packages (2024.5.0)\n"
     ]
    }
   ],
   "source": [
    "# use fsspec to read the document\n",
    "!pip install fsspec"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "import fsspec\n",
    "import os\n",
    "import time\n",
    "def get_local_file_metadata(file_path: str):\n",
    "    stat = os.stat(file_path)\n",
    "    return {\n",
    "        'size': stat.st_size,  # File size in bytes\n",
    "        'creation_date': time.ctime(stat.st_ctime),  # Creation time\n",
    "        'last_modified_date': time.ctime(stat.st_mtime)  # Last modification time\n",
    "    }\n",
    "\n",
    "\n",
    "def load_text_file(file_path: str) -> str:\n",
    "    \"\"\"\n",
    "    Loads a text file from the specified path using fsspec.\n",
    "\n",
    "    Args:\n",
    "        file_path (str): The path to the text file. This can be a local path or a URL for a supported file system.\n",
    "\n",
    "    # Example usage with a local file\n",
    "    local_file_path = 'file:///path/to/localfile.txt'\n",
    "    print(load_text_file(local_file_path))\n",
    "\n",
    "    # Example usage with an S3 file\n",
    "    s3_file_path = 's3://mybucket/myfile.txt'\n",
    "    print(load_text_file(s3_file_path))\n",
    "\n",
    "    # Example usage with a GCS file\n",
    "    gcs_file_path = 'gcs://mybucket/myfile.txt'\n",
    "    print(load_text_file(gcs_file_path))\n",
    "\n",
    "    # Example usage with an HTTP file\n",
    "    http_file_path = 'https://example.com/myfile.txt'\n",
    "    print(load_text_file(http_file_path))\n",
    "\n",
    "    Returns:\n",
    "        str: The content of the text file.\n",
    "    \"\"\"\n",
    "    with fsspec.open(file_path, 'r') as file:\n",
    "        content = file.read()\n",
    "    return content\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "What I Worked On\n",
      "\n",
      "February 2021\n",
      "\n",
      "Before college the two main things I worked on, outside of school, were writing and programming. I didn't write essays. I wrote what beginning writers were supposed to write then, and probably still are: short stories. My stories were awful. They had hardly any plot, just characters with strong feelings, which I imagined made them deep.\n",
      "\n",
      "The first programs I tried writing were on the IBM 1401 that our school district used for what was then called \"data processing.\" This was in 9th grade, so I was 13 or 14. The school district's 1401 happened to be in the basement of our junior high school, and my friend Rich Draves and I got permission to use it. It was like a mini Bond villain's lair down there, with all these alien-looking machines — CPU, disk drives, printer, card reader — sitting up on a raised floor under bright fluorescent lights.\n",
      "\n",
      "The language we used was an early version of Fortran. You had to type programs on punch cards, then stack them in t\n",
      "{'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}\n"
     ]
    }
   ],
   "source": [
    "text = load_text_file('paul_graham/paul_graham_essay.txt')\n",
    "file_metadata = get_local_file_metadata('paul_graham/paul_graham_essay.txt')\n",
    "print(text[:1000])\n",
    "print(file_metadata)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[Document(id=4e32f7b4-a82d-415b-a1e8-e63ef07e9ac6, text=\n",
      "\n",
      "What I Worked On\n",
      "\n",
      "February 2021\n",
      "\n",
      "Before college the two main things I worked on, outside of school, were writing and programming. I didn't write essays. I wrote what beginning writers were supposed to write then, and probably still are: short stories. My stories were awful. They had hardly any plot, just characters with strong feelings, which I imagined made them deep.\n",
      "\n",
      "The first programs I trie ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=16534)]\n",
      "DocumentSplitter(split_by=word, split_length=800, split_overlap=200)\n"
     ]
    }
   ],
   "source": [
    "# split the documents\n",
    "\n",
    "from lightrag.components.data_process import DocumentSplitter\n",
    "from lightrag.core.types import Document\n",
    "\n",
    "# sentence splitting is confusing, the length needs to be smaller\n",
    "metadata = {\"title\": \"Paul Graham's essay\", \"path\": \"data/paul_graham/paul_graham_essay.txt\"}\n",
    "metadata.update(file_metadata)\n",
    "documents = [Document(text = text, meta_data = metadata)]\n",
    "splitter = DocumentSplitter(split_by=\"word\", split_length=800, split_overlap=200)\n",
    "\n",
    "print(documents)\n",
    "print(splitter)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "20"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "token_limit = 16385\n",
    "\n",
    "# compute the maximum number of splitted_documents with split length = 800 and overlap = 200\n",
    "# total of 28 subdocuments now\n",
    "\n",
    "16385 // 800"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From the document structure, we can see the ``estimated_num_tokens=16534`` this will help us\n",
    "adapt our retriever."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Splitting documents:   0%|          | 0/1 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Splitting documents: 100%|██████████| 1/1 [00:00<00:00, 28.08it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Document(id=9e3a3e94-0fb1-4aaa-a9cf-006325349358, text=\n",
      "\n",
      "What I Worked On\n",
      "\n",
      "February 2021\n",
      "\n",
      "Before college the two main things I worked on, outside of school, were writing and programming. I didn't write essays. I wrote what beginning writers were supposed to write then, and probably still are: short stories. My stories were awful. They had hardly any plot, just characters with strong feelings, which I imagined made them deep.\n",
      "\n",
      "The first programs I trie ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=985, parent_doc_id=4e32f7b4-a82d-415b-a1e8-e63ef07e9ac6) 23\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "# split the document\n",
    "splitted_documents = splitter(documents = documents)\n",
    "print(splitted_documents[0], len(splitted_documents))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "LoggedLLMRetriever(\n",
      "  (generator): Generator(\n",
      "    model_kwargs={'model': 'gpt-3.5-turbo'}, \n",
      "    (prompt): Prompt(\n",
      "      template: <SYS>\n",
      "      Your are a retriever. Given a list of documents in the context, \\\n",
      "      you will retrieve a list of {{top_k}} indices(int) of the documents that are most relevant to the query. You will output a list as follows:\n",
      "      [<id from the most relevent with top_k options>]\n",
      "      <Documents>\n",
      "      {% for doc in documents %}\n",
      "      ```{{ loop.index - 1}}. {{doc}}```\n",
      "      {% endfor %}\n",
      "      </Documents>\n",
      "      </SYS>\n",
      "      Query: {{input_str}}\n",
      "      You:\n",
      "      , preset_prompt_kwargs: {'top_k': 2, 'documents': ['\\n\\nWhat I Worked On\\n\\nFebruary 2021\\n\\nBefore college the two main things I worked on, outside of school, were writing and programming. I didn\\'t write essays. I wrote what beginning writers were supposed to write then, and probably still are: short stories. My stories were awful. They had hardly any plot, just characters with strong feelings, which I imagined made them deep.\\n\\nThe first programs I tried writing were on the IBM 1401 that our school district used for what was then called \"data processing.\" This was in 9th grade, so I was 13 or 14. The school district\\'s 1401 happened to be in the basement of our junior high school, and my friend Rich Draves and I got permission to use it. It was like a mini Bond villain\\'s lair down there, with all these alien-looking machines — CPU, disk drives, printer, card reader — sitting up on a raised floor under bright fluorescent lights.\\n\\nThe language we used was an early version of Fortran. You had to type programs on punch cards, then stack them in the card reader and press a button to load the program into memory and run it. The result would ordinarily be to print something on the spectacularly loud printer.\\n\\nI was puzzled by the 1401. I couldn\\'t figure out what to do with it. And in retrospect there\\'s not much I could have done with it. The only form of input to programs was data stored on punched cards, and I didn\\'t have any data stored on punched cards. The only other option was to do things that didn\\'t rely on any input, like calculate approximations of pi, but I didn\\'t know enough math to do anything interesting of that type. So I\\'m not surprised I can\\'t remember any programs I wrote, because they can\\'t have done much. My clearest memory is of the moment I learned it was possible for programs not to terminate, when one of mine didn\\'t. On a machine without time-sharing, this was a social as well as a technical error, as the data center manager\\'s expression made clear.\\n\\nWith microcomputers, everything changed. Now you could have a computer sitting right in front of you, on a desk, that could respond to your keystrokes as it was running instead of just churning through a stack of punch cards and then stopping. [1]\\n\\nThe first of my friends to get a microcomputer built it himself. It was sold as a kit by Heathkit. I remember vividly how impressed and envious I felt watching him sitting in front of it, typing programs right into the computer.\\n\\nComputers were expensive in those days and it took me years of nagging before I convinced my father to buy one, a TRS-80, in about 1980. The gold standard then was the Apple II, but a TRS-80 was good enough. This was when I really started programming. I wrote simple games, a program to predict how high my model rockets would fly, and a word processor that my father used to write at least one book. There was only room in memory for about 2 pages of text, so he\\'d write 2 pages at a time and then print them out, but it was a lot better than a typewriter.\\n\\nThough I liked programming, I didn\\'t plan to study it in college. In college I was going to study philosophy, which sounded much more powerful. It seemed, to my naive high school self, to be the study of the ultimate truths, compared to which the things studied in other fields would be mere domain knowledge. What I discovered when I got to college was that the other fields took up so much of the space of ideas that there wasn\\'t much left for these supposed ultimate truths. All that seemed left for philosophy were edge cases that people in other fields felt could safely be ignored.\\n\\nI couldn\\'t have put this into words when I was 18. All I knew at the time was that I kept taking philosophy courses and they kept being boring. So I decided to switch to AI.\\n\\nAI was in the air in the mid 1980s, but there were two things especially that made me want to work on it: a novel by Heinlein called The Moon is a Harsh Mistress, which featured an intelligent computer called Mike, and a PBS documentary that showed Terry Winograd using SHRDLU. I haven\\'t tried rereading The Moon is a Harsh Mistress, so I don\\'t know how well it has aged, but when I read it I was drawn entirely into its world. It seemed only a matter of time before we\\'d have Mike, and when I saw Winograd using SHRDLU, it seemed like that time would be a few years at most. All you had to do was teach SHRDLU more words.\\n\\nThere weren\\'t ', 'was that the other fields took up so much of the space of ideas that there wasn\\'t much left for these supposed ultimate truths. All that seemed left for philosophy were edge cases that people in other fields felt could safely be ignored.\\n\\nI couldn\\'t have put this into words when I was 18. All I knew at the time was that I kept taking philosophy courses and they kept being boring. So I decided to switch to AI.\\n\\nAI was in the air in the mid 1980s, but there were two things especially that made me want to work on it: a novel by Heinlein called The Moon is a Harsh Mistress, which featured an intelligent computer called Mike, and a PBS documentary that showed Terry Winograd using SHRDLU. I haven\\'t tried rereading The Moon is a Harsh Mistress, so I don\\'t know how well it has aged, but when I read it I was drawn entirely into its world. It seemed only a matter of time before we\\'d have Mike, and when I saw Winograd using SHRDLU, it seemed like that time would be a few years at most. All you had to do was teach SHRDLU more words.\\n\\nThere weren\\'t any classes in AI at Cornell then, not even graduate classes, so I started trying to teach myself. Which meant learning Lisp, since in those days Lisp was regarded as the language of AI. The commonly used programming languages then were pretty primitive, and programmers\\' ideas correspondingly so. The default language at Cornell was a Pascal-like language called PL/I, and the situation was similar elsewhere. Learning Lisp expanded my concept of a program so fast that it was years before I started to have a sense of where the new limits were. This was more like it; this was what I had expected college to do. It wasn\\'t happening in a class, like it was supposed to, but that was ok. For the next couple years I was on a roll. I knew what I was going to do.\\n\\nFor my undergraduate thesis, I reverse-engineered SHRDLU. My God did I love working on that program. It was a pleasing bit of code, but what made it even more exciting was my belief — hard to imagine now, but not unique in 1985 — that it was already climbing the lower slopes of intelligence.\\n\\nI had gotten into a program at Cornell that didn\\'t make you choose a major. You could take whatever classes you liked, and choose whatever you liked to put on your degree. I of course chose \"Artificial Intelligence.\" When I got the actual physical diploma, I was dismayed to find that the quotes had been included, which made them read as scare-quotes. At the time this bothered me, but now it seems amusingly accurate, for reasons I was about to discover.\\n\\nI applied to 3 grad schools: MIT and Yale, which were renowned for AI at the time, and Harvard, which I\\'d visited because Rich Draves went there, and was also home to Bill Woods, who\\'d invented the type of parser I used in my SHRDLU clone. Only Harvard accepted me, so that was where I went.\\n\\nI don\\'t remember the moment it happened, or if there even was a specific moment, but during the first year of grad school I realized that AI, as practiced at the time, was a hoax. By which I mean the sort of AI in which a program that\\'s told \"the dog is sitting on the chair\" translates this into some formal representation and adds it to the list of things it knows.\\n\\nWhat these programs really showed was that there\\'s a subset of natural language that\\'s a formal language. But a very proper subset. It was clear that there was an unbridgeable gap between what they could do and actually understanding natural language. It was not, in fact, simply a matter of teaching SHRDLU more words. That whole way of doing AI, with explicit data structures representing concepts, was not going to work. Its brokenness did, as so often happens, generate a lot of opportunities to write papers about various band-aids that could be applied to it, but it was never going to get us Mike.\\n\\nSo I looked around to see what I could salvage from the wreckage of my plans, and there was Lisp. I knew from experience that Lisp was interesting for its own sake and not just for its association with AI, even though that was the main reason people cared about it at the time. So I decided to focus on Lisp. In fact, I decided to write a book about Lisp hacking. It\\'s scary to think how little I knew about Lisp hacking when I started writing that book. But there\\'s nothing like writing a book about something to help ', \"really showed was that there's a subset of natural language that's a formal language. But a very proper subset. It was clear that there was an unbridgeable gap between what they could do and actually understanding natural language. It was not, in fact, simply a matter of teaching SHRDLU more words. That whole way of doing AI, with explicit data structures representing concepts, was not going to work. Its brokenness did, as so often happens, generate a lot of opportunities to write papers about various band-aids that could be applied to it, but it was never going to get us Mike.\\n\\nSo I looked around to see what I could salvage from the wreckage of my plans, and there was Lisp. I knew from experience that Lisp was interesting for its own sake and not just for its association with AI, even though that was the main reason people cared about it at the time. So I decided to focus on Lisp. In fact, I decided to write a book about Lisp hacking. It's scary to think how little I knew about Lisp hacking when I started writing that book. But there's nothing like writing a book about something to help you learn it. The book, On Lisp, wasn't published till 1993, but I wrote much of it in grad school.\\n\\nComputer Science is an uneasy alliance between two halves, theory and systems. The theory people prove things, and the systems people build things. I wanted to build things. I had plenty of respect for theory — indeed, a sneaking suspicion that it was the more admirable of the two halves — but building things seemed so much more exciting.\\n\\nThe problem with systems work, though, was that it didn't last. Any program you wrote today, no matter how good, would be obsolete in a couple decades at best. People might mention your software in footnotes, but no one would actually use it. And indeed, it would seem very feeble work. Only people with a sense of the history of the field would even realize that, in its time, it had been good.\\n\\nThere were some surplus Xerox Dandelions floating around the computer lab at one point. Anyone who wanted one to play around with could have one. I was briefly tempted, but they were so slow by present standards; what was the point? No one else wanted one either, so off they went. That was what happened to systems work.\\n\\nI wanted not just to build things, but to build things that would last.\\n\\nIn this dissatisfied state I went in 1988 to visit Rich Draves at CMU, where he was in grad school. One day I went to visit the Carnegie Institute, where I'd spent a lot of time as a kid. While looking at a painting there I realized something that might seem obvious, but was a big surprise to me. There, right on the wall, was something you could make that would last. Paintings didn't become obsolete. Some of the best ones were hundreds of years old.\\n\\nAnd moreover this was something you could make a living doing. Not as easily as you could by writing software, of course, but I thought if you were really industrious and lived really cheaply, it had to be possible to make enough to survive. And as an artist you could be truly independent. You wouldn't have a boss, or even need to get research funding.\\n\\nI had always liked looking at paintings. Could I make them? I had no idea. I'd never imagined it was even possible. I knew intellectually that people made art — that it didn't just appear spontaneously — but it was as if the people who made it were a different species. They either lived long ago or were mysterious geniuses doing strange things in profiles in Life magazine. The idea of actually being able to make art, to put that verb before that noun, seemed almost miraculous.\\n\\nThat fall I started taking art classes at Harvard. Grad students could take classes in any department, and my advisor, Tom Cheatham, was very easy going. If he even knew about the strange classes I was taking, he never said anything.\\n\\nSo now I was in a PhD program in computer science, yet planning to be an artist, yet also genuinely in love with Lisp hacking and working away at On Lisp. In other words, like many a grad student, I was working energetically on multiple projects that were not my thesis.\\n\\nI didn't see a way out of this situation. I didn't want to drop out of grad school, but how else was I going to get out? I remember when my friend Robert Morris got kicked out of Cornell for writing the internet worm of 1988, I was envious that he'd found such a spectacular way \", 'didn\\'t just appear spontaneously — but it was as if the people who made it were a different species. They either lived long ago or were mysterious geniuses doing strange things in profiles in Life magazine. The idea of actually being able to make art, to put that verb before that noun, seemed almost miraculous.\\n\\nThat fall I started taking art classes at Harvard. Grad students could take classes in any department, and my advisor, Tom Cheatham, was very easy going. If he even knew about the strange classes I was taking, he never said anything.\\n\\nSo now I was in a PhD program in computer science, yet planning to be an artist, yet also genuinely in love with Lisp hacking and working away at On Lisp. In other words, like many a grad student, I was working energetically on multiple projects that were not my thesis.\\n\\nI didn\\'t see a way out of this situation. I didn\\'t want to drop out of grad school, but how else was I going to get out? I remember when my friend Robert Morris got kicked out of Cornell for writing the internet worm of 1988, I was envious that he\\'d found such a spectacular way to get out of grad school.\\n\\nThen one day in April 1990 a crack appeared in the wall. I ran into professor Cheatham and he asked if I was far enough along to graduate that June. I didn\\'t have a word of my dissertation written, but in what must have been the quickest bit of thinking in my life, I decided to take a shot at writing one in the 5 weeks or so that remained before the deadline, reusing parts of On Lisp where I could, and I was able to respond, with no perceptible delay \"Yes, I think so. I\\'ll give you something to read in a few days.\"\\n\\nI picked applications of continuations as the topic. In retrospect I should have written about macros and embedded languages. There\\'s a whole world there that\\'s barely been explored. But all I wanted was to get out of grad school, and my rapidly written dissertation sufficed, just barely.\\n\\nMeanwhile I was applying to art schools. I applied to two: RISD in the US, and the Accademia di Belli Arti in Florence, which, because it was the oldest art school, I imagined would be good. RISD accepted me, and I never heard back from the Accademia, so off to Providence I went.\\n\\nI\\'d applied for the BFA program at RISD, which meant in effect that I had to go to college again. This was not as strange as it sounds, because I was only 25, and art schools are full of people of different ages. RISD counted me as a transfer sophomore and said I had to do the foundation that summer. The foundation means the classes that everyone has to take in fundamental subjects like drawing, color, and design.\\n\\nToward the end of the summer I got a big surprise: a letter from the Accademia, which had been delayed because they\\'d sent it to Cambridge England instead of Cambridge Massachusetts, inviting me to take the entrance exam in Florence that fall. This was now only weeks away. My nice landlady let me leave my stuff in her attic. I had some money saved from consulting work I\\'d done in grad school; there was probably enough to last a year if I lived cheaply. Now all I had to do was learn Italian.\\n\\nOnly stranieri (foreigners) had to take this entrance exam. In retrospect it may well have been a way of excluding them, because there were so many stranieri attracted by the idea of studying art in Florence that the Italian students would otherwise have been outnumbered. I was in decent shape at painting and drawing from the RISD foundation that summer, but I still don\\'t know how I managed to pass the written exam. I remember that I answered the essay question by writing about Cezanne, and that I cranked up the intellectual level as high as I could to make the most of my limited vocabulary. [2]\\n\\nI\\'m only up to age 25 and already there are such conspicuous patterns. Here I was, yet again about to attend some august institution in the hopes of learning about some prestigious subject, and yet again about to be disappointed. The students and faculty in the painting department at the Accademia were the nicest people you could imagine, but they had long since arrived at an arrangement whereby the students wouldn\\'t require the faculty to teach anything, and in return the faculty wouldn\\'t require the students to learn anything. And at the same time all involved would adhere outwardly to the conventions of a 19th century atelier. We actually had one of those little stoves, fed with kindling, ', 'so many stranieri attracted by the idea of studying art in Florence that the Italian students would otherwise have been outnumbered. I was in decent shape at painting and drawing from the RISD foundation that summer, but I still don\\'t know how I managed to pass the written exam. I remember that I answered the essay question by writing about Cezanne, and that I cranked up the intellectual level as high as I could to make the most of my limited vocabulary. [2]\\n\\nI\\'m only up to age 25 and already there are such conspicuous patterns. Here I was, yet again about to attend some august institution in the hopes of learning about some prestigious subject, and yet again about to be disappointed. The students and faculty in the painting department at the Accademia were the nicest people you could imagine, but they had long since arrived at an arrangement whereby the students wouldn\\'t require the faculty to teach anything, and in return the faculty wouldn\\'t require the students to learn anything. And at the same time all involved would adhere outwardly to the conventions of a 19th century atelier. We actually had one of those little stoves, fed with kindling, that you see in 19th century studio paintings, and a nude model sitting as close to it as possible without getting burned. Except hardly anyone else painted her besides me. The rest of the students spent their time chatting or occasionally trying to imitate things they\\'d seen in American art magazines.\\n\\nOur model turned out to live just down the street from me. She made a living from a combination of modelling and making fakes for a local antique dealer. She\\'d copy an obscure old painting out of a book, and then he\\'d take the copy and maltreat it to make it look old. [3]\\n\\nWhile I was a student at the Accademia I started painting still lives in my bedroom at night. These paintings were tiny, because the room was, and because I painted them on leftover scraps of canvas, which was all I could afford at the time. Painting still lives is different from painting people, because the subject, as its name suggests, can\\'t move. People can\\'t sit for more than about 15 minutes at a time, and when they do they don\\'t sit very still. So the traditional m.o. for painting people is to know how to paint a generic person, which you then modify to match the specific person you\\'re painting. Whereas a still life you can, if you want, copy pixel by pixel from what you\\'re seeing. You don\\'t want to stop there, of course, or you get merely photographic accuracy, and what makes a still life interesting is that it\\'s been through a head. You want to emphasize the visual cues that tell you, for example, that the reason the color changes suddenly at a certain point is that it\\'s the edge of an object. By subtly emphasizing such things you can make paintings that are more realistic than photographs not just in some metaphorical sense, but in the strict information-theoretic sense. [4]\\n\\nI liked painting still lives because I was curious about what I was seeing. In everyday life, we aren\\'t consciously aware of much we\\'re seeing. Most visual perception is handled by low-level processes that merely tell your brain \"that\\'s a water droplet\" without telling you details like where the lightest and darkest points are, or \"that\\'s a bush\" without telling you the shape and position of every leaf. This is a feature of brains, not a bug. In everyday life it would be distracting to notice every leaf on every bush. But when you have to paint something, you have to look more closely, and when you do there\\'s a lot to see. You can still be noticing new things after days of trying to paint something people usually take for granted, just as you can after days of trying to write an essay about something people usually take for granted.\\n\\nThis is not the only way to paint. I\\'m not 100% sure it\\'s even a good way to paint. But it seemed a good enough bet to be worth trying.\\n\\nOur teacher, professor Ulivi, was a nice guy. He could see I worked hard, and gave me a good grade, which he wrote down in a sort of passport each student had. But the Accademia wasn\\'t teaching me anything except Italian, and my money was running out, so at the end of the first year I went back to the US.\\n\\nI wanted to go back to RISD, but I was now broke and RISD was very expensive, so I decided to get a job for a year and then return to RISD the next fall. I got one at a company called Interleaf, which made ', 'to notice every leaf on every bush. But when you have to paint something, you have to look more closely, and when you do there\\'s a lot to see. You can still be noticing new things after days of trying to paint something people usually take for granted, just as you can after days of trying to write an essay about something people usually take for granted.\\n\\nThis is not the only way to paint. I\\'m not 100% sure it\\'s even a good way to paint. But it seemed a good enough bet to be worth trying.\\n\\nOur teacher, professor Ulivi, was a nice guy. He could see I worked hard, and gave me a good grade, which he wrote down in a sort of passport each student had. But the Accademia wasn\\'t teaching me anything except Italian, and my money was running out, so at the end of the first year I went back to the US.\\n\\nI wanted to go back to RISD, but I was now broke and RISD was very expensive, so I decided to get a job for a year and then return to RISD the next fall. I got one at a company called Interleaf, which made software for creating documents. You mean like Microsoft Word? Exactly. That was how I learned that low end software tends to eat high end software. But Interleaf still had a few years to live yet. [5]\\n\\nInterleaf had done something pretty bold. Inspired by Emacs, they\\'d added a scripting language, and even made the scripting language a dialect of Lisp. Now they wanted a Lisp hacker to write things in it. This was the closest thing I\\'ve had to a normal job, and I hereby apologize to my boss and coworkers, because I was a bad employee. Their Lisp was the thinnest icing on a giant C cake, and since I didn\\'t know C and didn\\'t want to learn it, I never understood most of the software. Plus I was terribly irresponsible. This was back when a programming job meant showing up every day during certain working hours. That seemed unnatural to me, and on this point the rest of the world is coming around to my way of thinking, but at the time it caused a lot of friction. Toward the end of the year I spent much of my time surreptitiously working on On Lisp, which I had by this time gotten a contract to publish.\\n\\nThe good part was that I got paid huge amounts of money, especially by art student standards. In Florence, after paying my part of the rent, my budget for everything else had been $7 a day. Now I was getting paid more than 4 times that every hour, even when I was just sitting in a meeting. By living cheaply I not only managed to save enough to go back to RISD, but also paid off my college loans.\\n\\nI learned some useful things at Interleaf, though they were mostly about what not to do. I learned that it\\'s better for technology companies to be run by product people than sales people (though sales is a real skill and people who are good at it are really good at it), that it leads to bugs when code is edited by too many people, that cheap office space is no bargain if it\\'s depressing, that planned meetings are inferior to corridor conversations, that big, bureaucratic customers are a dangerous source of money, and that there\\'s not much overlap between conventional office hours and the optimal time for hacking, or conventional offices and the optimal place for it.\\n\\nBut the most important thing I learned, and which I used in both Viaweb and Y Combinator, is that the low end eats the high end: that it\\'s good to be the \"entry level\" option, even though that will be less prestigious, because if you\\'re not, someone else will be, and will squash you against the ceiling. Which in turn means that prestige is a danger sign.\\n\\nWhen I left to go back to RISD the next fall, I arranged to do freelance work for the group that did projects for customers, and this was how I survived for the next several years. When I came back to visit for a project later on, someone told me about a new thing called HTML, which was, as he described it, a derivative of SGML. Markup language enthusiasts were an occupational hazard at Interleaf and I ignored him, but this HTML thing later became a big part of my life.\\n\\nIn the fall of 1992 I moved back to Providence to continue at RISD. The foundation had merely been intro stuff, and the Accademia had been a (very civilized) joke. Now I was going to see what real art school was like. But alas ', 'it.\\n\\nBut the most important thing I learned, and which I used in both Viaweb and Y Combinator, is that the low end eats the high end: that it\\'s good to be the \"entry level\" option, even though that will be less prestigious, because if you\\'re not, someone else will be, and will squash you against the ceiling. Which in turn means that prestige is a danger sign.\\n\\nWhen I left to go back to RISD the next fall, I arranged to do freelance work for the group that did projects for customers, and this was how I survived for the next several years. When I came back to visit for a project later on, someone told me about a new thing called HTML, which was, as he described it, a derivative of SGML. Markup language enthusiasts were an occupational hazard at Interleaf and I ignored him, but this HTML thing later became a big part of my life.\\n\\nIn the fall of 1992 I moved back to Providence to continue at RISD. The foundation had merely been intro stuff, and the Accademia had been a (very civilized) joke. Now I was going to see what real art school was like. But alas it was more like the Accademia than not. Better organized, certainly, and a lot more expensive, but it was now becoming clear that art school did not bear the same relationship to art that medical school bore to medicine. At least not the painting department. The textile department, which my next door neighbor belonged to, seemed to be pretty rigorous. No doubt illustration and architecture were too. But painting was post-rigorous. Painting students were supposed to express themselves, which to the more worldly ones meant to try to cook up some sort of distinctive signature style.\\n\\nA signature style is the visual equivalent of what in show business is known as a \"schtick\": something that immediately identifies the work as yours and no one else\\'s. For example, when you see a painting that looks like a certain kind of cartoon, you know it\\'s by Roy Lichtenstein. So if you see a big painting of this type hanging in the apartment of a hedge fund manager, you know he paid millions of dollars for it. That\\'s not always why artists have a signature style, but it\\'s usually why buyers pay a lot for such work. [6]\\n\\nThere were plenty of earnest students too: kids who \"could draw\" in high school, and now had come to what was supposed to be the best art school in the country, to learn to draw even better. They tended to be confused and demoralized by what they found at RISD, but they kept going, because painting was what they did. I was not one of the kids who could draw in high school, but at RISD I was definitely closer to their tribe than the tribe of signature style seekers.\\n\\nI learned a lot in the color class I took at RISD, but otherwise I was basically teaching myself to paint, and I could do that for free. So in 1993 I dropped out. I hung around Providence for a bit, and then my college friend Nancy Parmet did me a big favor. A rent-controlled apartment in a building her mother owned in New York was becoming vacant. Did I want it? It wasn\\'t much more than my current place, and New York was supposed to be where the artists were. So yes, I wanted it! [7]\\n\\nAsterix comics begin by zooming in on a tiny corner of Roman Gaul that turns out not to be controlled by the Romans. You can do something similar on a map of New York City: if you zoom in on the Upper East Side, there\\'s a tiny corner that\\'s not rich, or at least wasn\\'t in 1993. It\\'s called Yorkville, and that was my new home. Now I was a New York artist — in the strictly technical sense of making paintings and living in New York.\\n\\nI was nervous about money, because I could sense that Interleaf was on the way down. Freelance Lisp hacking work was very rare, and I didn\\'t want to have to program in another language, which in those days would have meant C++ if I was lucky. So with my unerring nose for financial opportunity, I decided to write another book on Lisp. This would be a popular book, the sort of book that could be used as a textbook. I imagined myself living frugally off the royalties and spending all my time painting. (The painting on the cover of this book, ANSI Common Lisp, is one that I painted around this time.)\\n\\nThe best thing about New York for me was the presence of Idelle and Julian Weber. Idelle Weber was a painter, one of the early ', 'Romans. You can do something similar on a map of New York City: if you zoom in on the Upper East Side, there\\'s a tiny corner that\\'s not rich, or at least wasn\\'t in 1993. It\\'s called Yorkville, and that was my new home. Now I was a New York artist — in the strictly technical sense of making paintings and living in New York.\\n\\nI was nervous about money, because I could sense that Interleaf was on the way down. Freelance Lisp hacking work was very rare, and I didn\\'t want to have to program in another language, which in those days would have meant C++ if I was lucky. So with my unerring nose for financial opportunity, I decided to write another book on Lisp. This would be a popular book, the sort of book that could be used as a textbook. I imagined myself living frugally off the royalties and spending all my time painting. (The painting on the cover of this book, ANSI Common Lisp, is one that I painted around this time.)\\n\\nThe best thing about New York for me was the presence of Idelle and Julian Weber. Idelle Weber was a painter, one of the early photorealists, and I\\'d taken her painting class at Harvard. I\\'ve never known a teacher more beloved by her students. Large numbers of former students kept in touch with her, including me. After I moved to New York I became her de facto studio assistant.\\n\\nShe liked to paint on big, square canvases, 4 to 5 feet on a side. One day in late 1994 as I was stretching one of these monsters there was something on the radio about a famous fund manager. He wasn\\'t that much older than me, and was super rich. The thought suddenly occurred to me: why don\\'t I become rich? Then I\\'ll be able to work on whatever I want.\\n\\nMeanwhile I\\'d been hearing more and more about this new thing called the World Wide Web. Robert Morris showed it to me when I visited him in Cambridge, where he was now in grad school at Harvard. It seemed to me that the web would be a big deal. I\\'d seen what graphical user interfaces had done for the popularity of microcomputers. It seemed like the web would do the same for the internet.\\n\\nIf I wanted to get rich, here was the next train leaving the station. I was right about that part. What I got wrong was the idea. I decided we should start a company to put art galleries online. I can\\'t honestly say, after reading so many Y Combinator applications, that this was the worst startup idea ever, but it was up there. Art galleries didn\\'t want to be online, and still don\\'t, not the fancy ones. That\\'s not how they sell. I wrote some software to generate web sites for galleries, and Robert wrote some to resize images and set up an http server to serve the pages. Then we tried to sign up galleries. To call this a difficult sale would be an understatement. It was difficult to give away. A few galleries let us make sites for them for free, but none paid us.\\n\\nThen some online stores started to appear, and I realized that except for the order buttons they were identical to the sites we\\'d been generating for galleries. This impressive-sounding thing called an \"internet storefront\" was something we already knew how to build.\\n\\nSo in the summer of 1995, after I submitted the camera-ready copy of ANSI Common Lisp to the publishers, we started trying to write software to build online stores. At first this was going to be normal desktop software, which in those days meant Windows software. That was an alarming prospect, because neither of us knew how to write Windows software or wanted to learn. We lived in the Unix world. But we decided we\\'d at least try writing a prototype store builder on Unix. Robert wrote a shopping cart, and I wrote a new site generator for stores — in Lisp, of course.\\n\\nWe were working out of Robert\\'s apartment in Cambridge. His roommate was away for big chunks of time, during which I got to sleep in his room. For some reason there was no bed frame or sheets, just a mattress on the floor. One morning as I was lying on this mattress I had an idea that made me sit up like a capital L. What if we ran the software on the server, and let users control it by clicking on links? Then we\\'d never have to write anything to run on users\\' computers. We could generate the sites on the same server we\\'d serve them from. Users wouldn\\'t need anything more than a browser.\\n\\nThis kind of software, known as a web app, ', 'online stores. At first this was going to be normal desktop software, which in those days meant Windows software. That was an alarming prospect, because neither of us knew how to write Windows software or wanted to learn. We lived in the Unix world. But we decided we\\'d at least try writing a prototype store builder on Unix. Robert wrote a shopping cart, and I wrote a new site generator for stores — in Lisp, of course.\\n\\nWe were working out of Robert\\'s apartment in Cambridge. His roommate was away for big chunks of time, during which I got to sleep in his room. For some reason there was no bed frame or sheets, just a mattress on the floor. One morning as I was lying on this mattress I had an idea that made me sit up like a capital L. What if we ran the software on the server, and let users control it by clicking on links? Then we\\'d never have to write anything to run on users\\' computers. We could generate the sites on the same server we\\'d serve them from. Users wouldn\\'t need anything more than a browser.\\n\\nThis kind of software, known as a web app, is common now, but at the time it wasn\\'t clear that it was even possible. To find out, we decided to try making a version of our store builder that you could control through the browser. A couple days later, on August 12, we had one that worked. The UI was horrible, but it proved you could build a whole store through the browser, without any client software or typing anything into the command line on the server.\\n\\nNow we felt like we were really onto something. I had visions of a whole new generation of software working this way. You wouldn\\'t need versions, or ports, or any of that crap. At Interleaf there had been a whole group called Release Engineering that seemed to be at least as big as the group that actually wrote the software. Now you could just update the software right on the server.\\n\\nWe started a new company we called Viaweb, after the fact that our software worked via the web, and we got $10,000 in seed funding from Idelle\\'s husband Julian. In return for that and doing the initial legal work and giving us business advice, we gave him 10% of the company. Ten years later this deal became the model for Y Combinator\\'s. We knew founders needed something like this, because we\\'d needed it ourselves.\\n\\nAt this stage I had a negative net worth, because the thousand dollars or so I had in the bank was more than counterbalanced by what I owed the government in taxes. (Had I diligently set aside the proper proportion of the money I\\'d made consulting for Interleaf? No, I had not.) So although Robert had his graduate student stipend, I needed that seed funding to live on.\\n\\nWe originally hoped to launch in September, but we got more ambitious about the software as we worked on it. Eventually we managed to build a WYSIWYG site builder, in the sense that as you were creating pages, they looked exactly like the static ones that would be generated later, except that instead of leading to static pages, the links all referred to closures stored in a hash table on the server.\\n\\nIt helped to have studied art, because the main goal of an online store builder is to make users look legit, and the key to looking legit is high production values. If you get page layouts and fonts and colors right, you can make a guy running a store out of his bedroom look more legit than a big company.\\n\\n(If you\\'re curious why my site looks so old-fashioned, it\\'s because it\\'s still made with this software. It may look clunky today, but in 1996 it was the last word in slick.)\\n\\nIn September, Robert rebelled. \"We\\'ve been working on this for a month,\" he said, \"and it\\'s still not done.\" This is funny in retrospect, because he would still be working on it almost 3 years later. But I decided it might be prudent to recruit more programmers, and I asked Robert who else in grad school with him was really good. He recommended Trevor Blackwell, which surprised me at first, because at that point I knew Trevor mainly for his plan to reduce everything in his life to a stack of notecards, which he carried around with him. But Rtm was right, as usual. Trevor turned out to be a frighteningly effective hacker.\\n\\nIt was a lot of fun working with Robert and Trevor. They\\'re the two most independent-minded people I know, and in completely different ways. If you could see inside Rtm\\'s brain it would look like a colonial New England church, ', 'can make a guy running a store out of his bedroom look more legit than a big company.\\n\\n(If you\\'re curious why my site looks so old-fashioned, it\\'s because it\\'s still made with this software. It may look clunky today, but in 1996 it was the last word in slick.)\\n\\nIn September, Robert rebelled. \"We\\'ve been working on this for a month,\" he said, \"and it\\'s still not done.\" This is funny in retrospect, because he would still be working on it almost 3 years later. But I decided it might be prudent to recruit more programmers, and I asked Robert who else in grad school with him was really good. He recommended Trevor Blackwell, which surprised me at first, because at that point I knew Trevor mainly for his plan to reduce everything in his life to a stack of notecards, which he carried around with him. But Rtm was right, as usual. Trevor turned out to be a frighteningly effective hacker.\\n\\nIt was a lot of fun working with Robert and Trevor. They\\'re the two most independent-minded people I know, and in completely different ways. If you could see inside Rtm\\'s brain it would look like a colonial New England church, and if you could see inside Trevor\\'s it would look like the worst excesses of Austrian Rococo.\\n\\nWe opened for business, with 6 stores, in January 1996. It was just as well we waited a few months, because although we worried we were late, we were actually almost fatally early. There was a lot of talk in the press then about ecommerce, but not many people actually wanted online stores. [8]\\n\\nThere were three main parts to the software: the editor, which people used to build sites and which I wrote, the shopping cart, which Robert wrote, and the manager, which kept track of orders and statistics, and which Trevor wrote. In its time, the editor was one of the best general-purpose site builders. I kept the code tight and didn\\'t have to integrate with any other software except Robert\\'s and Trevor\\'s, so it was quite fun to work on. If all I\\'d had to do was work on this software, the next 3 years would have been the easiest of my life. Unfortunately I had to do a lot more, all of it stuff I was worse at than programming, and the next 3 years were instead the most stressful.\\n\\nThere were a lot of startups making ecommerce software in the second half of the 90s. We were determined to be the Microsoft Word, not the Interleaf. Which meant being easy to use and inexpensive. It was lucky for us that we were poor, because that caused us to make Viaweb even more inexpensive than we realized. We charged $100 a month for a small store and $300 a month for a big one. This low price was a big attraction, and a constant thorn in the sides of competitors, but it wasn\\'t because of some clever insight that we set the price low. We had no idea what businesses paid for things. $300 a month seemed like a lot of money to us.\\n\\nWe did a lot of things right by accident like that. For example, we did what\\'s now called \"doing things that don\\'t scale,\" although at the time we would have described it as \"being so lame that we\\'re driven to the most desperate measures to get users.\" The most common of which was building stores for them. This seemed particularly humiliating, since the whole raison d\\'etre of our software was that people could use it to make their own stores. But anything to get users.\\n\\nWe learned a lot more about retail than we wanted to know. For example, that if you could only have a small image of a man\\'s shirt (and all images were small then by present standards), it was better to have a closeup of the collar than a picture of the whole shirt. The reason I remember learning this was that it meant I had to rescan about 30 images of men\\'s shirts. My first set of scans were so beautiful too.\\n\\nThough this felt wrong, it was exactly the right thing to be doing. Building stores for users taught us about retail, and about how it felt to use our software. I was initially both mystified and repelled by \"business\" and thought we needed a \"business person\" to be in charge of it, but once we started to get users, I was converted, in much the same way I was converted to fatherhood once I had kids. Whatever users wanted, I was all theirs. Maybe one day we\\'d have so many users that I couldn\\'t scan their images for them, but in the meantime there was nothing more important to do.\\n\\nAnother thing I didn\\'t get ', 'stores. But anything to get users.\\n\\nWe learned a lot more about retail than we wanted to know. For example, that if you could only have a small image of a man\\'s shirt (and all images were small then by present standards), it was better to have a closeup of the collar than a picture of the whole shirt. The reason I remember learning this was that it meant I had to rescan about 30 images of men\\'s shirts. My first set of scans were so beautiful too.\\n\\nThough this felt wrong, it was exactly the right thing to be doing. Building stores for users taught us about retail, and about how it felt to use our software. I was initially both mystified and repelled by \"business\" and thought we needed a \"business person\" to be in charge of it, but once we started to get users, I was converted, in much the same way I was converted to fatherhood once I had kids. Whatever users wanted, I was all theirs. Maybe one day we\\'d have so many users that I couldn\\'t scan their images for them, but in the meantime there was nothing more important to do.\\n\\nAnother thing I didn\\'t get at the time is that growth rate is the ultimate test of a startup. Our growth rate was fine. We had about 70 stores at the end of 1996 and about 500 at the end of 1997. I mistakenly thought the thing that mattered was the absolute number of users. And that is the thing that matters in the sense that that\\'s how much money you\\'re making, and if you\\'re not making enough, you might go out of business. But in the long term the growth rate takes care of the absolute number. If we\\'d been a startup I was advising at Y Combinator, I would have said: Stop being so stressed out, because you\\'re doing fine. You\\'re growing 7x a year. Just don\\'t hire too many more people and you\\'ll soon be profitable, and then you\\'ll control your own destiny.\\n\\nAlas I hired lots more people, partly because our investors wanted me to, and partly because that\\'s what startups did during the Internet Bubble. A company with just a handful of employees would have seemed amateurish. So we didn\\'t reach breakeven until about when Yahoo bought us in the summer of 1998. Which in turn meant we were at the mercy of investors for the entire life of the company. And since both we and our investors were noobs at startups, the result was a mess even by startup standards.\\n\\nIt was a huge relief when Yahoo bought us. In principle our Viaweb stock was valuable. It was a share in a business that was profitable and growing rapidly. But it didn\\'t feel very valuable to me; I had no idea how to value a business, but I was all too keenly aware of the near-death experiences we seemed to have every few months. Nor had I changed my grad student lifestyle significantly since we started. So when Yahoo bought us it felt like going from rags to riches. Since we were going to California, I bought a car, a yellow 1998 VW GTI. I remember thinking that its leather seats alone were by far the most luxurious thing I owned.\\n\\nThe next year, from the summer of 1998 to the summer of 1999, must have been the least productive of my life. I didn\\'t realize it at the time, but I was worn out from the effort and stress of running Viaweb. For a while after I got to California I tried to continue my usual m.o. of programming till 3 in the morning, but fatigue combined with Yahoo\\'s prematurely aged culture and grim cube farm in Santa Clara gradually dragged me down. After a few months it felt disconcertingly like working at Interleaf.\\n\\nYahoo had given us a lot of options when they bought us. At the time I thought Yahoo was so overvalued that they\\'d never be worth anything, but to my astonishment the stock went up 5x in the next year. I hung on till the first chunk of options vested, then in the summer of 1999 I left. It had been so long since I\\'d painted anything that I\\'d half forgotten why I was doing this. My brain had been entirely full of software and men\\'s shirts for 4 years. But I had done this to get rich so I could paint, I reminded myself, and now I was rich, so I should go paint.\\n\\nWhen I said I was leaving, my boss at Yahoo had a long conversation with me about my plans. I told him all about the kinds of pictures I wanted to paint. At the time I was touched that he took such an interest ', \"tried to continue my usual m.o. of programming till 3 in the morning, but fatigue combined with Yahoo's prematurely aged culture and grim cube farm in Santa Clara gradually dragged me down. After a few months it felt disconcertingly like working at Interleaf.\\n\\nYahoo had given us a lot of options when they bought us. At the time I thought Yahoo was so overvalued that they'd never be worth anything, but to my astonishment the stock went up 5x in the next year. I hung on till the first chunk of options vested, then in the summer of 1999 I left. It had been so long since I'd painted anything that I'd half forgotten why I was doing this. My brain had been entirely full of software and men's shirts for 4 years. But I had done this to get rich so I could paint, I reminded myself, and now I was rich, so I should go paint.\\n\\nWhen I said I was leaving, my boss at Yahoo had a long conversation with me about my plans. I told him all about the kinds of pictures I wanted to paint. At the time I was touched that he took such an interest in me. Now I realize it was because he thought I was lying. My options at that point were worth about $2 million a month. If I was leaving that kind of money on the table, it could only be to go and start some new startup, and if I did, I might take people with me. This was the height of the Internet Bubble, and Yahoo was ground zero of it. My boss was at that moment a billionaire. Leaving then to start a new startup must have seemed to him an insanely, and yet also plausibly, ambitious plan.\\n\\nBut I really was quitting to paint, and I started immediately. There was no time to lose. I'd already burned 4 years getting rich. Now when I talk to founders who are leaving after selling their companies, my advice is always the same: take a vacation. That's what I should have done, just gone off somewhere and done nothing for a month or two, but the idea never occurred to me.\\n\\nSo I tried to paint, but I just didn't seem to have any energy or ambition. Part of the problem was that I didn't know many people in California. I'd compounded this problem by buying a house up in the Santa Cruz Mountains, with a beautiful view but miles from anywhere. I stuck it out for a few more months, then in desperation I went back to New York, where unless you understand about rent control you'll be surprised to hear I still had my apartment, sealed up like a tomb of my old life. Idelle was in New York at least, and there were other people trying to paint there, even though I didn't know any of them.\\n\\nWhen I got back to New York I resumed my old life, except now I was rich. It was as weird as it sounds. I resumed all my old patterns, except now there were doors where there hadn't been. Now when I was tired of walking, all I had to do was raise my hand, and (unless it was raining) a taxi would stop to pick me up. Now when I walked past charming little restaurants I could go in and order lunch. It was exciting for a while. Painting started to go better. I experimented with a new kind of still life where I'd paint one painting in the old way, then photograph it and print it, blown up, on canvas, and then use that as the underpainting for a second still life, painted from the same objects (which hopefully hadn't rotted yet).\\n\\nMeanwhile I looked for an apartment to buy. Now I could actually choose what neighborhood to live in. Where, I asked myself and various real estate agents, is the Cambridge of New York? Aided by occasional visits to actual Cambridge, I gradually realized there wasn't one. Huh.\\n\\nAround this time, in the spring of 2000, I had an idea. It was clear from our experience with Viaweb that web apps were the future. Why not build a web app for making web apps? Why not let people edit code on our server through the browser, and then host the resulting applications for them? [9] You could run all sorts of services on the servers that these applications could use just by making an API call: making and receiving phone calls, manipulating images, taking credit card payments, etc.\\n\\nI got so excited about this idea that I couldn't think about anything else. It seemed obvious that this was the future. I didn't particularly want to start another company, but it was clear that \", 'photograph it and print it, blown up, on canvas, and then use that as the underpainting for a second still life, painted from the same objects (which hopefully hadn\\'t rotted yet).\\n\\nMeanwhile I looked for an apartment to buy. Now I could actually choose what neighborhood to live in. Where, I asked myself and various real estate agents, is the Cambridge of New York? Aided by occasional visits to actual Cambridge, I gradually realized there wasn\\'t one. Huh.\\n\\nAround this time, in the spring of 2000, I had an idea. It was clear from our experience with Viaweb that web apps were the future. Why not build a web app for making web apps? Why not let people edit code on our server through the browser, and then host the resulting applications for them? [9] You could run all sorts of services on the servers that these applications could use just by making an API call: making and receiving phone calls, manipulating images, taking credit card payments, etc.\\n\\nI got so excited about this idea that I couldn\\'t think about anything else. It seemed obvious that this was the future. I didn\\'t particularly want to start another company, but it was clear that this idea would have to be embodied as one, so I decided to move to Cambridge and start it. I hoped to lure Robert into working on it with me, but there I ran into a hitch. Robert was now a postdoc at MIT, and though he\\'d made a lot of money the last time I\\'d lured him into working on one of my schemes, it had also been a huge time sink. So while he agreed that it sounded like a plausible idea, he firmly refused to work on it.\\n\\nHmph. Well, I\\'d do it myself then. I recruited Dan Giffin, who had worked for Viaweb, and two undergrads who wanted summer jobs, and we got to work trying to build what it\\'s now clear is about twenty companies and several open source projects worth of software. The language for defining applications would of course be a dialect of Lisp. But I wasn\\'t so naive as to assume I could spring an overt Lisp on a general audience; we\\'d hide the parentheses, like Dylan did.\\n\\nBy then there was a name for the kind of company Viaweb was, an \"application service provider,\" or ASP. This name didn\\'t last long before it was replaced by \"software as a service,\" but it was current for long enough that I named this new company after it: it was going to be called Aspra.\\n\\nI started working on the application builder, Dan worked on network infrastructure, and the two undergrads worked on the first two services (images and phone calls). But about halfway through the summer I realized I really didn\\'t want to run a company — especially not a big one, which it was looking like this would have to be. I\\'d only started Viaweb because I needed the money. Now that I didn\\'t need money anymore, why was I doing this? If this vision had to be realized as a company, then screw the vision. I\\'d build a subset that could be done as an open source project.\\n\\nMuch to my surprise, the time I spent working on this stuff was not wasted after all. After we started Y Combinator, I would often encounter startups working on parts of this new architecture, and it was very useful to have spent so much time thinking about it and even trying to write some of it.\\n\\nThe subset I would build as an open source project was the new Lisp, whose parentheses I now wouldn\\'t even have to hide. A lot of Lisp hackers dream of building a new Lisp, partly because one of the distinctive features of the language is that it has dialects, and partly, I think, because we have in our minds a Platonic form of Lisp that all existing dialects fall short of. I certainly did. So at the end of the summer Dan and I switched to working on this new dialect of Lisp, which I called Arc, in a house I bought in Cambridge.\\n\\nThe following spring, lightning struck. I was invited to give a talk at a Lisp conference, so I gave one about how we\\'d used Lisp at Viaweb. Afterward I put a postscript file of this talk online, on paulgraham.com, which I\\'d created years before using Viaweb but had never used for anything. In one day it got 30,000 page views. What on earth had happened? The referring urls showed that someone had posted it on Slashdot. [10]\\n\\nWow, I thought, there\\'s an audience. If I write something and put it on the web, anyone can read it. That may seem obvious now, but it was surprising then. In the print ', \"new Lisp, whose parentheses I now wouldn't even have to hide. A lot of Lisp hackers dream of building a new Lisp, partly because one of the distinctive features of the language is that it has dialects, and partly, I think, because we have in our minds a Platonic form of Lisp that all existing dialects fall short of. I certainly did. So at the end of the summer Dan and I switched to working on this new dialect of Lisp, which I called Arc, in a house I bought in Cambridge.\\n\\nThe following spring, lightning struck. I was invited to give a talk at a Lisp conference, so I gave one about how we'd used Lisp at Viaweb. Afterward I put a postscript file of this talk online, on paulgraham.com, which I'd created years before using Viaweb but had never used for anything. In one day it got 30,000 page views. What on earth had happened? The referring urls showed that someone had posted it on Slashdot. [10]\\n\\nWow, I thought, there's an audience. If I write something and put it on the web, anyone can read it. That may seem obvious now, but it was surprising then. In the print era there was a narrow channel to readers, guarded by fierce monsters known as editors. The only way to get an audience for anything you wrote was to get it published as a book, or in a newspaper or magazine. Now anyone could publish anything.\\n\\nThis had been possible in principle since 1993, but not many people had realized it yet. I had been intimately involved with building the infrastructure of the web for most of that time, and a writer as well, and it had taken me 8 years to realize it. Even then it took me several years to understand the implications. It meant there would be a whole new generation of essays. [11]\\n\\nIn the print era, the channel for publishing essays had been vanishingly small. Except for a few officially anointed thinkers who went to the right parties in New York, the only people allowed to publish essays were specialists writing about their specialties. There were so many essays that had never been written, because there had been no way to publish them. Now they could be, and I was going to write them. [12]\\n\\nI've worked on several different things, but to the extent there was a turning point where I figured out what to work on, it was when I started publishing essays online. From then on I knew that whatever else I did, I'd always write essays too.\\n\\nI knew that online essays would be a marginal medium at first. Socially they'd seem more like rants posted by nutjobs on their GeoCities sites than the genteel and beautifully typeset compositions published in The New Yorker. But by this point I knew enough to find that encouraging instead of discouraging.\\n\\nOne of the most conspicuous patterns I've noticed in my life is how well it has worked, for me at least, to work on things that weren't prestigious. Still life has always been the least prestigious form of painting. Viaweb and Y Combinator both seemed lame when we started them. I still get the glassy eye from strangers when they ask what I'm writing, and I explain that it's an essay I'm going to publish on my web site. Even Lisp, though prestigious intellectually in something like the way Latin is, also seems about as hip.\\n\\nIt's not that unprestigious types of work are good per se. But when you find yourself drawn to some kind of work despite its current lack of prestige, it's a sign both that there's something real to be discovered there, and that you have the right kind of motives. Impure motives are a big danger for the ambitious. If anything is going to lead you astray, it will be the desire to impress people. So while working on things that aren't prestigious doesn't guarantee you're on the right track, it at least guarantees you're not on the most common type of wrong one.\\n\\nOver the next several years I wrote lots of essays about all kinds of different topics. O'Reilly reprinted a collection of them as a book, called Hackers & Painters after one of the essays in it. I also worked on spam filters, and did some more painting. I used to have dinners for a group of friends every thursday night, which taught me how to cook for groups. And I bought another building in Cambridge, a former candy factory (and later, twas said, porn studio), to use as an office.\\n\\nOne night in October 2003 there was a big party at my house. It was a clever idea of my friend Maria Daniels, who was one of the thursday diners. Three separate hosts \", 'current lack of prestige, it\\'s a sign both that there\\'s something real to be discovered there, and that you have the right kind of motives. Impure motives are a big danger for the ambitious. If anything is going to lead you astray, it will be the desire to impress people. So while working on things that aren\\'t prestigious doesn\\'t guarantee you\\'re on the right track, it at least guarantees you\\'re not on the most common type of wrong one.\\n\\nOver the next several years I wrote lots of essays about all kinds of different topics. O\\'Reilly reprinted a collection of them as a book, called Hackers & Painters after one of the essays in it. I also worked on spam filters, and did some more painting. I used to have dinners for a group of friends every thursday night, which taught me how to cook for groups. And I bought another building in Cambridge, a former candy factory (and later, twas said, porn studio), to use as an office.\\n\\nOne night in October 2003 there was a big party at my house. It was a clever idea of my friend Maria Daniels, who was one of the thursday diners. Three separate hosts would all invite their friends to one party. So for every guest, two thirds of the other guests would be people they didn\\'t know but would probably like. One of the guests was someone I didn\\'t know but would turn out to like a lot: a woman called Jessica Livingston. A couple days later I asked her out.\\n\\nJessica was in charge of marketing at a Boston investment bank. This bank thought it understood startups, but over the next year, as she met friends of mine from the startup world, she was surprised how different reality was. And how colorful their stories were. So she decided to compile a book of interviews with startup founders.\\n\\nWhen the bank had financial problems and she had to fire half her staff, she started looking for a new job. In early 2005 she interviewed for a marketing job at a Boston VC firm. It took them weeks to make up their minds, and during this time I started telling her about all the things that needed to be fixed about venture capital. They should make a larger number of smaller investments instead of a handful of giant ones, they should be funding younger, more technical founders instead of MBAs, they should let the founders remain as CEO, and so on.\\n\\nOne of my tricks for writing essays had always been to give talks. The prospect of having to stand up in front of a group of people and tell them something that won\\'t waste their time is a great spur to the imagination. When the Harvard Computer Society, the undergrad computer club, asked me to give a talk, I decided I would tell them how to start a startup. Maybe they\\'d be able to avoid the worst of the mistakes we\\'d made.\\n\\nSo I gave this talk, in the course of which I told them that the best sources of seed funding were successful startup founders, because then they\\'d be sources of advice too. Whereupon it seemed they were all looking expectantly at me. Horrified at the prospect of having my inbox flooded by business plans (if I\\'d only known), I blurted out \"But not me!\" and went on with the talk. But afterward it occurred to me that I should really stop procrastinating about angel investing. I\\'d been meaning to since Yahoo bought us, and now it was 7 years later and I still hadn\\'t done one angel investment.\\n\\nMeanwhile I had been scheming with Robert and Trevor about projects we could work on together. I missed working with them, and it seemed like there had to be something we could collaborate on.\\n\\nAs Jessica and I were walking home from dinner on March 11, at the corner of Garden and Walker streets, these three threads converged. Screw the VCs who were taking so long to make up their minds. We\\'d start our own investment firm and actually implement the ideas we\\'d been talking about. I\\'d fund it, and Jessica could quit her job and work for it, and we\\'d get Robert and Trevor as partners too. [13]\\n\\nOnce again, ignorance worked in our favor. We had no idea how to be angel investors, and in Boston in 2005 there were no Ron Conways to learn from. So we just made what seemed like the obvious choices, and some of the things we did turned out to be novel.\\n\\nThere are multiple components to Y Combinator, and we didn\\'t figure them all out at once. The part we got first was to be an angel firm. In those days, those two words didn\\'t go together. There were VC firms, which ', 'one angel investment.\\n\\nMeanwhile I had been scheming with Robert and Trevor about projects we could work on together. I missed working with them, and it seemed like there had to be something we could collaborate on.\\n\\nAs Jessica and I were walking home from dinner on March 11, at the corner of Garden and Walker streets, these three threads converged. Screw the VCs who were taking so long to make up their minds. We\\'d start our own investment firm and actually implement the ideas we\\'d been talking about. I\\'d fund it, and Jessica could quit her job and work for it, and we\\'d get Robert and Trevor as partners too. [13]\\n\\nOnce again, ignorance worked in our favor. We had no idea how to be angel investors, and in Boston in 2005 there were no Ron Conways to learn from. So we just made what seemed like the obvious choices, and some of the things we did turned out to be novel.\\n\\nThere are multiple components to Y Combinator, and we didn\\'t figure them all out at once. The part we got first was to be an angel firm. In those days, those two words didn\\'t go together. There were VC firms, which were organized companies with people whose job it was to make investments, but they only did big, million dollar investments. And there were angels, who did smaller investments, but these were individuals who were usually focused on other things and made investments on the side. And neither of them helped founders enough in the beginning. We knew how helpless founders were in some respects, because we remembered how helpless we\\'d been. For example, one thing Julian had done for us that seemed to us like magic was to get us set up as a company. We were fine writing fairly difficult software, but actually getting incorporated, with bylaws and stock and all that stuff, how on earth did you do that? Our plan was not only to make seed investments, but to do for startups everything Julian had done for us.\\n\\nYC was not organized as a fund. It was cheap enough to run that we funded it with our own money. That went right by 99% of readers, but professional investors are thinking \"Wow, that means they got all the returns.\" But once again, this was not due to any particular insight on our part. We didn\\'t know how VC firms were organized. It never occurred to us to try to raise a fund, and if it had, we wouldn\\'t have known where to start. [14]\\n\\nThe most distinctive thing about YC is the batch model: to fund a bunch of startups all at once, twice a year, and then to spend three months focusing intensively on trying to help them. That part we discovered by accident, not merely implicitly but explicitly due to our ignorance about investing. We needed to get experience as investors. What better way, we thought, than to fund a whole bunch of startups at once? We knew undergrads got temporary jobs at tech companies during the summer. Why not organize a summer program where they\\'d start startups instead? We wouldn\\'t feel guilty for being in a sense fake investors, because they would in a similar sense be fake founders. So while we probably wouldn\\'t make much money out of it, we\\'d at least get to practice being investors on them, and they for their part would probably have a more interesting summer than they would working at Microsoft.\\n\\nWe\\'d use the building I owned in Cambridge as our headquarters. We\\'d all have dinner there once a week — on tuesdays, since I was already cooking for the thursday diners on thursdays — and after dinner we\\'d bring in experts on startups to give talks.\\n\\nWe knew undergrads were deciding then about summer jobs, so in a matter of days we cooked up something we called the Summer Founders Program, and I posted an announcement on my site, inviting undergrads to apply. I had never imagined that writing essays would be a way to get \"deal flow,\" as investors call it, but it turned out to be the perfect source. [15] We got 225 applications for the Summer Founders Program, and we were surprised to find that a lot of them were from people who\\'d already graduated, or were about to that spring. Already this SFP thing was starting to feel more serious than we\\'d intended.\\n\\nWe invited about 20 of the 225 groups to interview in person, and from those we picked 8 to fund. They were an impressive group. That first batch included reddit, Justin Kan and Emmett Shear, who went on to found Twitch, Aaron Swartz, who had already helped write the RSS spec and would a few years later become a martyr for open access, and ']}, prompt_variables: ['top_k', 'documents', 'input_str']\n",
      "    )\n",
      "    (model_client): OpenAIClient()\n",
      "    (output_processors): ListParser()\n",
      "  )\n",
      ")\n",
      "Prompt:\n",
      "<SYS>\n",
      "Your are a retriever. Given a list of documents in the context, \\\n",
      "you will retrieve a list of 2 indices(int) of the documents that are most relevant to the query. You will output a list as follows:\n",
      "[<id from the most relevent with top_k options>]\n",
      "<Documents>\n",
      "```0. \n",
      "\n",
      "What I Worked On\n",
      "\n",
      "February 2021\n",
      "\n",
      "Before college the two main things I worked on, outside of school, were writing and programming. I didn't write essays. I wrote what beginning writers were supposed to write then, and probably still are: short stories. My stories were awful. They had hardly any plot, just characters with strong feelings, which I imagined made them deep.\n",
      "\n",
      "The first programs I tried writing were on the IBM 1401 that our school district used for what was then called \"data processing.\" This was in 9th grade, so I was 13 or 14. The school district's 1401 happened to be in the basement of our junior high school, and my friend Rich Draves and I got permission to use it. It was like a mini Bond villain's lair down there, with all these alien-looking machines — CPU, disk drives, printer, card reader — sitting up on a raised floor under bright fluorescent lights.\n",
      "\n",
      "The language we used was an early version of Fortran. You had to type programs on punch cards, then stack them in the card reader and press a button to load the program into memory and run it. The result would ordinarily be to print something on the spectacularly loud printer.\n",
      "\n",
      "I was puzzled by the 1401. I couldn't figure out what to do with it. And in retrospect there's not much I could have done with it. The only form of input to programs was data stored on punched cards, and I didn't have any data stored on punched cards. The only other option was to do things that didn't rely on any input, like calculate approximations of pi, but I didn't know enough math to do anything interesting of that type. So I'm not surprised I can't remember any programs I wrote, because they can't have done much. My clearest memory is of the moment I learned it was possible for programs not to terminate, when one of mine didn't. On a machine without time-sharing, this was a social as well as a technical error, as the data center manager's expression made clear.\n",
      "\n",
      "With microcomputers, everything changed. Now you could have a computer sitting right in front of you, on a desk, that could respond to your keystrokes as it was running instead of just churning through a stack of punch cards and then stopping. [1]\n",
      "\n",
      "The first of my friends to get a microcomputer built it himself. It was sold as a kit by Heathkit. I remember vividly how impressed and envious I felt watching him sitting in front of it, typing programs right into the computer.\n",
      "\n",
      "Computers were expensive in those days and it took me years of nagging before I convinced my father to buy one, a TRS-80, in about 1980. The gold standard then was the Apple II, but a TRS-80 was good enough. This was when I really started programming. I wrote simple games, a program to predict how high my model rockets would fly, and a word processor that my father used to write at least one book. There was only room in memory for about 2 pages of text, so he'd write 2 pages at a time and then print them out, but it was a lot better than a typewriter.\n",
      "\n",
      "Though I liked programming, I didn't plan to study it in college. In college I was going to study philosophy, which sounded much more powerful. It seemed, to my naive high school self, to be the study of the ultimate truths, compared to which the things studied in other fields would be mere domain knowledge. What I discovered when I got to college was that the other fields took up so much of the space of ideas that there wasn't much left for these supposed ultimate truths. All that seemed left for philosophy were edge cases that people in other fields felt could safely be ignored.\n",
      "\n",
      "I couldn't have put this into words when I was 18. All I knew at the time was that I kept taking philosophy courses and they kept being boring. So I decided to switch to AI.\n",
      "\n",
      "AI was in the air in the mid 1980s, but there were two things especially that made me want to work on it: a novel by Heinlein called The Moon is a Harsh Mistress, which featured an intelligent computer called Mike, and a PBS documentary that showed Terry Winograd using SHRDLU. I haven't tried rereading The Moon is a Harsh Mistress, so I don't know how well it has aged, but when I read it I was drawn entirely into its world. It seemed only a matter of time before we'd have Mike, and when I saw Winograd using SHRDLU, it seemed like that time would be a few years at most. All you had to do was teach SHRDLU more words.\n",
      "\n",
      "There weren't ```\n",
      "```1. was that the other fields took up so much of the space of ideas that there wasn't much left for these supposed ultimate truths. All that seemed left for philosophy were edge cases that people in other fields felt could safely be ignored.\n",
      "\n",
      "I couldn't have put this into words when I was 18. All I knew at the time was that I kept taking philosophy courses and they kept being boring. So I decided to switch to AI.\n",
      "\n",
      "AI was in the air in the mid 1980s, but there were two things especially that made me want to work on it: a novel by Heinlein called The Moon is a Harsh Mistress, which featured an intelligent computer called Mike, and a PBS documentary that showed Terry Winograd using SHRDLU. I haven't tried rereading The Moon is a Harsh Mistress, so I don't know how well it has aged, but when I read it I was drawn entirely into its world. It seemed only a matter of time before we'd have Mike, and when I saw Winograd using SHRDLU, it seemed like that time would be a few years at most. All you had to do was teach SHRDLU more words.\n",
      "\n",
      "There weren't any classes in AI at Cornell then, not even graduate classes, so I started trying to teach myself. Which meant learning Lisp, since in those days Lisp was regarded as the language of AI. The commonly used programming languages then were pretty primitive, and programmers' ideas correspondingly so. The default language at Cornell was a Pascal-like language called PL/I, and the situation was similar elsewhere. Learning Lisp expanded my concept of a program so fast that it was years before I started to have a sense of where the new limits were. This was more like it; this was what I had expected college to do. It wasn't happening in a class, like it was supposed to, but that was ok. For the next couple years I was on a roll. I knew what I was going to do.\n",
      "\n",
      "For my undergraduate thesis, I reverse-engineered SHRDLU. My God did I love working on that program. It was a pleasing bit of code, but what made it even more exciting was my belief — hard to imagine now, but not unique in 1985 — that it was already climbing the lower slopes of intelligence.\n",
      "\n",
      "I had gotten into a program at Cornell that didn't make you choose a major. You could take whatever classes you liked, and choose whatever you liked to put on your degree. I of course chose \"Artificial Intelligence.\" When I got the actual physical diploma, I was dismayed to find that the quotes had been included, which made them read as scare-quotes. At the time this bothered me, but now it seems amusingly accurate, for reasons I was about to discover.\n",
      "\n",
      "I applied to 3 grad schools: MIT and Yale, which were renowned for AI at the time, and Harvard, which I'd visited because Rich Draves went there, and was also home to Bill Woods, who'd invented the type of parser I used in my SHRDLU clone. Only Harvard accepted me, so that was where I went.\n",
      "\n",
      "I don't remember the moment it happened, or if there even was a specific moment, but during the first year of grad school I realized that AI, as practiced at the time, was a hoax. By which I mean the sort of AI in which a program that's told \"the dog is sitting on the chair\" translates this into some formal representation and adds it to the list of things it knows.\n",
      "\n",
      "What these programs really showed was that there's a subset of natural language that's a formal language. But a very proper subset. It was clear that there was an unbridgeable gap between what they could do and actually understanding natural language. It was not, in fact, simply a matter of teaching SHRDLU more words. That whole way of doing AI, with explicit data structures representing concepts, was not going to work. Its brokenness did, as so often happens, generate a lot of opportunities to write papers about various band-aids that could be applied to it, but it was never going to get us Mike.\n",
      "\n",
      "So I looked around to see what I could salvage from the wreckage of my plans, and there was Lisp. I knew from experience that Lisp was interesting for its own sake and not just for its association with AI, even though that was the main reason people cared about it at the time. So I decided to focus on Lisp. In fact, I decided to write a book about Lisp hacking. It's scary to think how little I knew about Lisp hacking when I started writing that book. But there's nothing like writing a book about something to help ```\n",
      "```2. really showed was that there's a subset of natural language that's a formal language. But a very proper subset. It was clear that there was an unbridgeable gap between what they could do and actually understanding natural language. It was not, in fact, simply a matter of teaching SHRDLU more words. That whole way of doing AI, with explicit data structures representing concepts, was not going to work. Its brokenness did, as so often happens, generate a lot of opportunities to write papers about various band-aids that could be applied to it, but it was never going to get us Mike.\n",
      "\n",
      "So I looked around to see what I could salvage from the wreckage of my plans, and there was Lisp. I knew from experience that Lisp was interesting for its own sake and not just for its association with AI, even though that was the main reason people cared about it at the time. So I decided to focus on Lisp. In fact, I decided to write a book about Lisp hacking. It's scary to think how little I knew about Lisp hacking when I started writing that book. But there's nothing like writing a book about something to help you learn it. The book, On Lisp, wasn't published till 1993, but I wrote much of it in grad school.\n",
      "\n",
      "Computer Science is an uneasy alliance between two halves, theory and systems. The theory people prove things, and the systems people build things. I wanted to build things. I had plenty of respect for theory — indeed, a sneaking suspicion that it was the more admirable of the two halves — but building things seemed so much more exciting.\n",
      "\n",
      "The problem with systems work, though, was that it didn't last. Any program you wrote today, no matter how good, would be obsolete in a couple decades at best. People might mention your software in footnotes, but no one would actually use it. And indeed, it would seem very feeble work. Only people with a sense of the history of the field would even realize that, in its time, it had been good.\n",
      "\n",
      "There were some surplus Xerox Dandelions floating around the computer lab at one point. Anyone who wanted one to play around with could have one. I was briefly tempted, but they were so slow by present standards; what was the point? No one else wanted one either, so off they went. That was what happened to systems work.\n",
      "\n",
      "I wanted not just to build things, but to build things that would last.\n",
      "\n",
      "In this dissatisfied state I went in 1988 to visit Rich Draves at CMU, where he was in grad school. One day I went to visit the Carnegie Institute, where I'd spent a lot of time as a kid. While looking at a painting there I realized something that might seem obvious, but was a big surprise to me. There, right on the wall, was something you could make that would last. Paintings didn't become obsolete. Some of the best ones were hundreds of years old.\n",
      "\n",
      "And moreover this was something you could make a living doing. Not as easily as you could by writing software, of course, but I thought if you were really industrious and lived really cheaply, it had to be possible to make enough to survive. And as an artist you could be truly independent. You wouldn't have a boss, or even need to get research funding.\n",
      "\n",
      "I had always liked looking at paintings. Could I make them? I had no idea. I'd never imagined it was even possible. I knew intellectually that people made art — that it didn't just appear spontaneously — but it was as if the people who made it were a different species. They either lived long ago or were mysterious geniuses doing strange things in profiles in Life magazine. The idea of actually being able to make art, to put that verb before that noun, seemed almost miraculous.\n",
      "\n",
      "That fall I started taking art classes at Harvard. Grad students could take classes in any department, and my advisor, Tom Cheatham, was very easy going. If he even knew about the strange classes I was taking, he never said anything.\n",
      "\n",
      "So now I was in a PhD program in computer science, yet planning to be an artist, yet also genuinely in love with Lisp hacking and working away at On Lisp. In other words, like many a grad student, I was working energetically on multiple projects that were not my thesis.\n",
      "\n",
      "I didn't see a way out of this situation. I didn't want to drop out of grad school, but how else was I going to get out? I remember when my friend Robert Morris got kicked out of Cornell for writing the internet worm of 1988, I was envious that he'd found such a spectacular way ```\n",
      "```3. didn't just appear spontaneously — but it was as if the people who made it were a different species. They either lived long ago or were mysterious geniuses doing strange things in profiles in Life magazine. The idea of actually being able to make art, to put that verb before that noun, seemed almost miraculous.\n",
      "\n",
      "That fall I started taking art classes at Harvard. Grad students could take classes in any department, and my advisor, Tom Cheatham, was very easy going. If he even knew about the strange classes I was taking, he never said anything.\n",
      "\n",
      "So now I was in a PhD program in computer science, yet planning to be an artist, yet also genuinely in love with Lisp hacking and working away at On Lisp. In other words, like many a grad student, I was working energetically on multiple projects that were not my thesis.\n",
      "\n",
      "I didn't see a way out of this situation. I didn't want to drop out of grad school, but how else was I going to get out? I remember when my friend Robert Morris got kicked out of Cornell for writing the internet worm of 1988, I was envious that he'd found such a spectacular way to get out of grad school.\n",
      "\n",
      "Then one day in April 1990 a crack appeared in the wall. I ran into professor Cheatham and he asked if I was far enough along to graduate that June. I didn't have a word of my dissertation written, but in what must have been the quickest bit of thinking in my life, I decided to take a shot at writing one in the 5 weeks or so that remained before the deadline, reusing parts of On Lisp where I could, and I was able to respond, with no perceptible delay \"Yes, I think so. I'll give you something to read in a few days.\"\n",
      "\n",
      "I picked applications of continuations as the topic. In retrospect I should have written about macros and embedded languages. There's a whole world there that's barely been explored. But all I wanted was to get out of grad school, and my rapidly written dissertation sufficed, just barely.\n",
      "\n",
      "Meanwhile I was applying to art schools. I applied to two: RISD in the US, and the Accademia di Belli Arti in Florence, which, because it was the oldest art school, I imagined would be good. RISD accepted me, and I never heard back from the Accademia, so off to Providence I went.\n",
      "\n",
      "I'd applied for the BFA program at RISD, which meant in effect that I had to go to college again. This was not as strange as it sounds, because I was only 25, and art schools are full of people of different ages. RISD counted me as a transfer sophomore and said I had to do the foundation that summer. The foundation means the classes that everyone has to take in fundamental subjects like drawing, color, and design.\n",
      "\n",
      "Toward the end of the summer I got a big surprise: a letter from the Accademia, which had been delayed because they'd sent it to Cambridge England instead of Cambridge Massachusetts, inviting me to take the entrance exam in Florence that fall. This was now only weeks away. My nice landlady let me leave my stuff in her attic. I had some money saved from consulting work I'd done in grad school; there was probably enough to last a year if I lived cheaply. Now all I had to do was learn Italian.\n",
      "\n",
      "Only stranieri (foreigners) had to take this entrance exam. In retrospect it may well have been a way of excluding them, because there were so many stranieri attracted by the idea of studying art in Florence that the Italian students would otherwise have been outnumbered. I was in decent shape at painting and drawing from the RISD foundation that summer, but I still don't know how I managed to pass the written exam. I remember that I answered the essay question by writing about Cezanne, and that I cranked up the intellectual level as high as I could to make the most of my limited vocabulary. [2]\n",
      "\n",
      "I'm only up to age 25 and already there are such conspicuous patterns. Here I was, yet again about to attend some august institution in the hopes of learning about some prestigious subject, and yet again about to be disappointed. The students and faculty in the painting department at the Accademia were the nicest people you could imagine, but they had long since arrived at an arrangement whereby the students wouldn't require the faculty to teach anything, and in return the faculty wouldn't require the students to learn anything. And at the same time all involved would adhere outwardly to the conventions of a 19th century atelier. We actually had one of those little stoves, fed with kindling, ```\n",
      "```4. so many stranieri attracted by the idea of studying art in Florence that the Italian students would otherwise have been outnumbered. I was in decent shape at painting and drawing from the RISD foundation that summer, but I still don't know how I managed to pass the written exam. I remember that I answered the essay question by writing about Cezanne, and that I cranked up the intellectual level as high as I could to make the most of my limited vocabulary. [2]\n",
      "\n",
      "I'm only up to age 25 and already there are such conspicuous patterns. Here I was, yet again about to attend some august institution in the hopes of learning about some prestigious subject, and yet again about to be disappointed. The students and faculty in the painting department at the Accademia were the nicest people you could imagine, but they had long since arrived at an arrangement whereby the students wouldn't require the faculty to teach anything, and in return the faculty wouldn't require the students to learn anything. And at the same time all involved would adhere outwardly to the conventions of a 19th century atelier. We actually had one of those little stoves, fed with kindling, that you see in 19th century studio paintings, and a nude model sitting as close to it as possible without getting burned. Except hardly anyone else painted her besides me. The rest of the students spent their time chatting or occasionally trying to imitate things they'd seen in American art magazines.\n",
      "\n",
      "Our model turned out to live just down the street from me. She made a living from a combination of modelling and making fakes for a local antique dealer. She'd copy an obscure old painting out of a book, and then he'd take the copy and maltreat it to make it look old. [3]\n",
      "\n",
      "While I was a student at the Accademia I started painting still lives in my bedroom at night. These paintings were tiny, because the room was, and because I painted them on leftover scraps of canvas, which was all I could afford at the time. Painting still lives is different from painting people, because the subject, as its name suggests, can't move. People can't sit for more than about 15 minutes at a time, and when they do they don't sit very still. So the traditional m.o. for painting people is to know how to paint a generic person, which you then modify to match the specific person you're painting. Whereas a still life you can, if you want, copy pixel by pixel from what you're seeing. You don't want to stop there, of course, or you get merely photographic accuracy, and what makes a still life interesting is that it's been through a head. You want to emphasize the visual cues that tell you, for example, that the reason the color changes suddenly at a certain point is that it's the edge of an object. By subtly emphasizing such things you can make paintings that are more realistic than photographs not just in some metaphorical sense, but in the strict information-theoretic sense. [4]\n",
      "\n",
      "I liked painting still lives because I was curious about what I was seeing. In everyday life, we aren't consciously aware of much we're seeing. Most visual perception is handled by low-level processes that merely tell your brain \"that's a water droplet\" without telling you details like where the lightest and darkest points are, or \"that's a bush\" without telling you the shape and position of every leaf. This is a feature of brains, not a bug. In everyday life it would be distracting to notice every leaf on every bush. But when you have to paint something, you have to look more closely, and when you do there's a lot to see. You can still be noticing new things after days of trying to paint something people usually take for granted, just as you can after days of trying to write an essay about something people usually take for granted.\n",
      "\n",
      "This is not the only way to paint. I'm not 100% sure it's even a good way to paint. But it seemed a good enough bet to be worth trying.\n",
      "\n",
      "Our teacher, professor Ulivi, was a nice guy. He could see I worked hard, and gave me a good grade, which he wrote down in a sort of passport each student had. But the Accademia wasn't teaching me anything except Italian, and my money was running out, so at the end of the first year I went back to the US.\n",
      "\n",
      "I wanted to go back to RISD, but I was now broke and RISD was very expensive, so I decided to get a job for a year and then return to RISD the next fall. I got one at a company called Interleaf, which made ```\n",
      "```5. to notice every leaf on every bush. But when you have to paint something, you have to look more closely, and when you do there's a lot to see. You can still be noticing new things after days of trying to paint something people usually take for granted, just as you can after days of trying to write an essay about something people usually take for granted.\n",
      "\n",
      "This is not the only way to paint. I'm not 100% sure it's even a good way to paint. But it seemed a good enough bet to be worth trying.\n",
      "\n",
      "Our teacher, professor Ulivi, was a nice guy. He could see I worked hard, and gave me a good grade, which he wrote down in a sort of passport each student had. But the Accademia wasn't teaching me anything except Italian, and my money was running out, so at the end of the first year I went back to the US.\n",
      "\n",
      "I wanted to go back to RISD, but I was now broke and RISD was very expensive, so I decided to get a job for a year and then return to RISD the next fall. I got one at a company called Interleaf, which made software for creating documents. You mean like Microsoft Word? Exactly. That was how I learned that low end software tends to eat high end software. But Interleaf still had a few years to live yet. [5]\n",
      "\n",
      "Interleaf had done something pretty bold. Inspired by Emacs, they'd added a scripting language, and even made the scripting language a dialect of Lisp. Now they wanted a Lisp hacker to write things in it. This was the closest thing I've had to a normal job, and I hereby apologize to my boss and coworkers, because I was a bad employee. Their Lisp was the thinnest icing on a giant C cake, and since I didn't know C and didn't want to learn it, I never understood most of the software. Plus I was terribly irresponsible. This was back when a programming job meant showing up every day during certain working hours. That seemed unnatural to me, and on this point the rest of the world is coming around to my way of thinking, but at the time it caused a lot of friction. Toward the end of the year I spent much of my time surreptitiously working on On Lisp, which I had by this time gotten a contract to publish.\n",
      "\n",
      "The good part was that I got paid huge amounts of money, especially by art student standards. In Florence, after paying my part of the rent, my budget for everything else had been $7 a day. Now I was getting paid more than 4 times that every hour, even when I was just sitting in a meeting. By living cheaply I not only managed to save enough to go back to RISD, but also paid off my college loans.\n",
      "\n",
      "I learned some useful things at Interleaf, though they were mostly about what not to do. I learned that it's better for technology companies to be run by product people than sales people (though sales is a real skill and people who are good at it are really good at it), that it leads to bugs when code is edited by too many people, that cheap office space is no bargain if it's depressing, that planned meetings are inferior to corridor conversations, that big, bureaucratic customers are a dangerous source of money, and that there's not much overlap between conventional office hours and the optimal time for hacking, or conventional offices and the optimal place for it.\n",
      "\n",
      "But the most important thing I learned, and which I used in both Viaweb and Y Combinator, is that the low end eats the high end: that it's good to be the \"entry level\" option, even though that will be less prestigious, because if you're not, someone else will be, and will squash you against the ceiling. Which in turn means that prestige is a danger sign.\n",
      "\n",
      "When I left to go back to RISD the next fall, I arranged to do freelance work for the group that did projects for customers, and this was how I survived for the next several years. When I came back to visit for a project later on, someone told me about a new thing called HTML, which was, as he described it, a derivative of SGML. Markup language enthusiasts were an occupational hazard at Interleaf and I ignored him, but this HTML thing later became a big part of my life.\n",
      "\n",
      "In the fall of 1992 I moved back to Providence to continue at RISD. The foundation had merely been intro stuff, and the Accademia had been a (very civilized) joke. Now I was going to see what real art school was like. But alas ```\n",
      "```6. it.\n",
      "\n",
      "But the most important thing I learned, and which I used in both Viaweb and Y Combinator, is that the low end eats the high end: that it's good to be the \"entry level\" option, even though that will be less prestigious, because if you're not, someone else will be, and will squash you against the ceiling. Which in turn means that prestige is a danger sign.\n",
      "\n",
      "When I left to go back to RISD the next fall, I arranged to do freelance work for the group that did projects for customers, and this was how I survived for the next several years. When I came back to visit for a project later on, someone told me about a new thing called HTML, which was, as he described it, a derivative of SGML. Markup language enthusiasts were an occupational hazard at Interleaf and I ignored him, but this HTML thing later became a big part of my life.\n",
      "\n",
      "In the fall of 1992 I moved back to Providence to continue at RISD. The foundation had merely been intro stuff, and the Accademia had been a (very civilized) joke. Now I was going to see what real art school was like. But alas it was more like the Accademia than not. Better organized, certainly, and a lot more expensive, but it was now becoming clear that art school did not bear the same relationship to art that medical school bore to medicine. At least not the painting department. The textile department, which my next door neighbor belonged to, seemed to be pretty rigorous. No doubt illustration and architecture were too. But painting was post-rigorous. Painting students were supposed to express themselves, which to the more worldly ones meant to try to cook up some sort of distinctive signature style.\n",
      "\n",
      "A signature style is the visual equivalent of what in show business is known as a \"schtick\": something that immediately identifies the work as yours and no one else's. For example, when you see a painting that looks like a certain kind of cartoon, you know it's by Roy Lichtenstein. So if you see a big painting of this type hanging in the apartment of a hedge fund manager, you know he paid millions of dollars for it. That's not always why artists have a signature style, but it's usually why buyers pay a lot for such work. [6]\n",
      "\n",
      "There were plenty of earnest students too: kids who \"could draw\" in high school, and now had come to what was supposed to be the best art school in the country, to learn to draw even better. They tended to be confused and demoralized by what they found at RISD, but they kept going, because painting was what they did. I was not one of the kids who could draw in high school, but at RISD I was definitely closer to their tribe than the tribe of signature style seekers.\n",
      "\n",
      "I learned a lot in the color class I took at RISD, but otherwise I was basically teaching myself to paint, and I could do that for free. So in 1993 I dropped out. I hung around Providence for a bit, and then my college friend Nancy Parmet did me a big favor. A rent-controlled apartment in a building her mother owned in New York was becoming vacant. Did I want it? It wasn't much more than my current place, and New York was supposed to be where the artists were. So yes, I wanted it! [7]\n",
      "\n",
      "Asterix comics begin by zooming in on a tiny corner of Roman Gaul that turns out not to be controlled by the Romans. You can do something similar on a map of New York City: if you zoom in on the Upper East Side, there's a tiny corner that's not rich, or at least wasn't in 1993. It's called Yorkville, and that was my new home. Now I was a New York artist — in the strictly technical sense of making paintings and living in New York.\n",
      "\n",
      "I was nervous about money, because I could sense that Interleaf was on the way down. Freelance Lisp hacking work was very rare, and I didn't want to have to program in another language, which in those days would have meant C++ if I was lucky. So with my unerring nose for financial opportunity, I decided to write another book on Lisp. This would be a popular book, the sort of book that could be used as a textbook. I imagined myself living frugally off the royalties and spending all my time painting. (The painting on the cover of this book, ANSI Common Lisp, is one that I painted around this time.)\n",
      "\n",
      "The best thing about New York for me was the presence of Idelle and Julian Weber. Idelle Weber was a painter, one of the early ```\n",
      "```7. Romans. You can do something similar on a map of New York City: if you zoom in on the Upper East Side, there's a tiny corner that's not rich, or at least wasn't in 1993. It's called Yorkville, and that was my new home. Now I was a New York artist — in the strictly technical sense of making paintings and living in New York.\n",
      "\n",
      "I was nervous about money, because I could sense that Interleaf was on the way down. Freelance Lisp hacking work was very rare, and I didn't want to have to program in another language, which in those days would have meant C++ if I was lucky. So with my unerring nose for financial opportunity, I decided to write another book on Lisp. This would be a popular book, the sort of book that could be used as a textbook. I imagined myself living frugally off the royalties and spending all my time painting. (The painting on the cover of this book, ANSI Common Lisp, is one that I painted around this time.)\n",
      "\n",
      "The best thing about New York for me was the presence of Idelle and Julian Weber. Idelle Weber was a painter, one of the early photorealists, and I'd taken her painting class at Harvard. I've never known a teacher more beloved by her students. Large numbers of former students kept in touch with her, including me. After I moved to New York I became her de facto studio assistant.\n",
      "\n",
      "She liked to paint on big, square canvases, 4 to 5 feet on a side. One day in late 1994 as I was stretching one of these monsters there was something on the radio about a famous fund manager. He wasn't that much older than me, and was super rich. The thought suddenly occurred to me: why don't I become rich? Then I'll be able to work on whatever I want.\n",
      "\n",
      "Meanwhile I'd been hearing more and more about this new thing called the World Wide Web. Robert Morris showed it to me when I visited him in Cambridge, where he was now in grad school at Harvard. It seemed to me that the web would be a big deal. I'd seen what graphical user interfaces had done for the popularity of microcomputers. It seemed like the web would do the same for the internet.\n",
      "\n",
      "If I wanted to get rich, here was the next train leaving the station. I was right about that part. What I got wrong was the idea. I decided we should start a company to put art galleries online. I can't honestly say, after reading so many Y Combinator applications, that this was the worst startup idea ever, but it was up there. Art galleries didn't want to be online, and still don't, not the fancy ones. That's not how they sell. I wrote some software to generate web sites for galleries, and Robert wrote some to resize images and set up an http server to serve the pages. Then we tried to sign up galleries. To call this a difficult sale would be an understatement. It was difficult to give away. A few galleries let us make sites for them for free, but none paid us.\n",
      "\n",
      "Then some online stores started to appear, and I realized that except for the order buttons they were identical to the sites we'd been generating for galleries. This impressive-sounding thing called an \"internet storefront\" was something we already knew how to build.\n",
      "\n",
      "So in the summer of 1995, after I submitted the camera-ready copy of ANSI Common Lisp to the publishers, we started trying to write software to build online stores. At first this was going to be normal desktop software, which in those days meant Windows software. That was an alarming prospect, because neither of us knew how to write Windows software or wanted to learn. We lived in the Unix world. But we decided we'd at least try writing a prototype store builder on Unix. Robert wrote a shopping cart, and I wrote a new site generator for stores — in Lisp, of course.\n",
      "\n",
      "We were working out of Robert's apartment in Cambridge. His roommate was away for big chunks of time, during which I got to sleep in his room. For some reason there was no bed frame or sheets, just a mattress on the floor. One morning as I was lying on this mattress I had an idea that made me sit up like a capital L. What if we ran the software on the server, and let users control it by clicking on links? Then we'd never have to write anything to run on users' computers. We could generate the sites on the same server we'd serve them from. Users wouldn't need anything more than a browser.\n",
      "\n",
      "This kind of software, known as a web app, ```\n",
      "```8. online stores. At first this was going to be normal desktop software, which in those days meant Windows software. That was an alarming prospect, because neither of us knew how to write Windows software or wanted to learn. We lived in the Unix world. But we decided we'd at least try writing a prototype store builder on Unix. Robert wrote a shopping cart, and I wrote a new site generator for stores — in Lisp, of course.\n",
      "\n",
      "We were working out of Robert's apartment in Cambridge. His roommate was away for big chunks of time, during which I got to sleep in his room. For some reason there was no bed frame or sheets, just a mattress on the floor. One morning as I was lying on this mattress I had an idea that made me sit up like a capital L. What if we ran the software on the server, and let users control it by clicking on links? Then we'd never have to write anything to run on users' computers. We could generate the sites on the same server we'd serve them from. Users wouldn't need anything more than a browser.\n",
      "\n",
      "This kind of software, known as a web app, is common now, but at the time it wasn't clear that it was even possible. To find out, we decided to try making a version of our store builder that you could control through the browser. A couple days later, on August 12, we had one that worked. The UI was horrible, but it proved you could build a whole store through the browser, without any client software or typing anything into the command line on the server.\n",
      "\n",
      "Now we felt like we were really onto something. I had visions of a whole new generation of software working this way. You wouldn't need versions, or ports, or any of that crap. At Interleaf there had been a whole group called Release Engineering that seemed to be at least as big as the group that actually wrote the software. Now you could just update the software right on the server.\n",
      "\n",
      "We started a new company we called Viaweb, after the fact that our software worked via the web, and we got $10,000 in seed funding from Idelle's husband Julian. In return for that and doing the initial legal work and giving us business advice, we gave him 10% of the company. Ten years later this deal became the model for Y Combinator's. We knew founders needed something like this, because we'd needed it ourselves.\n",
      "\n",
      "At this stage I had a negative net worth, because the thousand dollars or so I had in the bank was more than counterbalanced by what I owed the government in taxes. (Had I diligently set aside the proper proportion of the money I'd made consulting for Interleaf? No, I had not.) So although Robert had his graduate student stipend, I needed that seed funding to live on.\n",
      "\n",
      "We originally hoped to launch in September, but we got more ambitious about the software as we worked on it. Eventually we managed to build a WYSIWYG site builder, in the sense that as you were creating pages, they looked exactly like the static ones that would be generated later, except that instead of leading to static pages, the links all referred to closures stored in a hash table on the server.\n",
      "\n",
      "It helped to have studied art, because the main goal of an online store builder is to make users look legit, and the key to looking legit is high production values. If you get page layouts and fonts and colors right, you can make a guy running a store out of his bedroom look more legit than a big company.\n",
      "\n",
      "(If you're curious why my site looks so old-fashioned, it's because it's still made with this software. It may look clunky today, but in 1996 it was the last word in slick.)\n",
      "\n",
      "In September, Robert rebelled. \"We've been working on this for a month,\" he said, \"and it's still not done.\" This is funny in retrospect, because he would still be working on it almost 3 years later. But I decided it might be prudent to recruit more programmers, and I asked Robert who else in grad school with him was really good. He recommended Trevor Blackwell, which surprised me at first, because at that point I knew Trevor mainly for his plan to reduce everything in his life to a stack of notecards, which he carried around with him. But Rtm was right, as usual. Trevor turned out to be a frighteningly effective hacker.\n",
      "\n",
      "It was a lot of fun working with Robert and Trevor. They're the two most independent-minded people I know, and in completely different ways. If you could see inside Rtm's brain it would look like a colonial New England church, ```\n",
      "```9. can make a guy running a store out of his bedroom look more legit than a big company.\n",
      "\n",
      "(If you're curious why my site looks so old-fashioned, it's because it's still made with this software. It may look clunky today, but in 1996 it was the last word in slick.)\n",
      "\n",
      "In September, Robert rebelled. \"We've been working on this for a month,\" he said, \"and it's still not done.\" This is funny in retrospect, because he would still be working on it almost 3 years later. But I decided it might be prudent to recruit more programmers, and I asked Robert who else in grad school with him was really good. He recommended Trevor Blackwell, which surprised me at first, because at that point I knew Trevor mainly for his plan to reduce everything in his life to a stack of notecards, which he carried around with him. But Rtm was right, as usual. Trevor turned out to be a frighteningly effective hacker.\n",
      "\n",
      "It was a lot of fun working with Robert and Trevor. They're the two most independent-minded people I know, and in completely different ways. If you could see inside Rtm's brain it would look like a colonial New England church, and if you could see inside Trevor's it would look like the worst excesses of Austrian Rococo.\n",
      "\n",
      "We opened for business, with 6 stores, in January 1996. It was just as well we waited a few months, because although we worried we were late, we were actually almost fatally early. There was a lot of talk in the press then about ecommerce, but not many people actually wanted online stores. [8]\n",
      "\n",
      "There were three main parts to the software: the editor, which people used to build sites and which I wrote, the shopping cart, which Robert wrote, and the manager, which kept track of orders and statistics, and which Trevor wrote. In its time, the editor was one of the best general-purpose site builders. I kept the code tight and didn't have to integrate with any other software except Robert's and Trevor's, so it was quite fun to work on. If all I'd had to do was work on this software, the next 3 years would have been the easiest of my life. Unfortunately I had to do a lot more, all of it stuff I was worse at than programming, and the next 3 years were instead the most stressful.\n",
      "\n",
      "There were a lot of startups making ecommerce software in the second half of the 90s. We were determined to be the Microsoft Word, not the Interleaf. Which meant being easy to use and inexpensive. It was lucky for us that we were poor, because that caused us to make Viaweb even more inexpensive than we realized. We charged $100 a month for a small store and $300 a month for a big one. This low price was a big attraction, and a constant thorn in the sides of competitors, but it wasn't because of some clever insight that we set the price low. We had no idea what businesses paid for things. $300 a month seemed like a lot of money to us.\n",
      "\n",
      "We did a lot of things right by accident like that. For example, we did what's now called \"doing things that don't scale,\" although at the time we would have described it as \"being so lame that we're driven to the most desperate measures to get users.\" The most common of which was building stores for them. This seemed particularly humiliating, since the whole raison d'etre of our software was that people could use it to make their own stores. But anything to get users.\n",
      "\n",
      "We learned a lot more about retail than we wanted to know. For example, that if you could only have a small image of a man's shirt (and all images were small then by present standards), it was better to have a closeup of the collar than a picture of the whole shirt. The reason I remember learning this was that it meant I had to rescan about 30 images of men's shirts. My first set of scans were so beautiful too.\n",
      "\n",
      "Though this felt wrong, it was exactly the right thing to be doing. Building stores for users taught us about retail, and about how it felt to use our software. I was initially both mystified and repelled by \"business\" and thought we needed a \"business person\" to be in charge of it, but once we started to get users, I was converted, in much the same way I was converted to fatherhood once I had kids. Whatever users wanted, I was all theirs. Maybe one day we'd have so many users that I couldn't scan their images for them, but in the meantime there was nothing more important to do.\n",
      "\n",
      "Another thing I didn't get ```\n",
      "```10. stores. But anything to get users.\n",
      "\n",
      "We learned a lot more about retail than we wanted to know. For example, that if you could only have a small image of a man's shirt (and all images were small then by present standards), it was better to have a closeup of the collar than a picture of the whole shirt. The reason I remember learning this was that it meant I had to rescan about 30 images of men's shirts. My first set of scans were so beautiful too.\n",
      "\n",
      "Though this felt wrong, it was exactly the right thing to be doing. Building stores for users taught us about retail, and about how it felt to use our software. I was initially both mystified and repelled by \"business\" and thought we needed a \"business person\" to be in charge of it, but once we started to get users, I was converted, in much the same way I was converted to fatherhood once I had kids. Whatever users wanted, I was all theirs. Maybe one day we'd have so many users that I couldn't scan their images for them, but in the meantime there was nothing more important to do.\n",
      "\n",
      "Another thing I didn't get at the time is that growth rate is the ultimate test of a startup. Our growth rate was fine. We had about 70 stores at the end of 1996 and about 500 at the end of 1997. I mistakenly thought the thing that mattered was the absolute number of users. And that is the thing that matters in the sense that that's how much money you're making, and if you're not making enough, you might go out of business. But in the long term the growth rate takes care of the absolute number. If we'd been a startup I was advising at Y Combinator, I would have said: Stop being so stressed out, because you're doing fine. You're growing 7x a year. Just don't hire too many more people and you'll soon be profitable, and then you'll control your own destiny.\n",
      "\n",
      "Alas I hired lots more people, partly because our investors wanted me to, and partly because that's what startups did during the Internet Bubble. A company with just a handful of employees would have seemed amateurish. So we didn't reach breakeven until about when Yahoo bought us in the summer of 1998. Which in turn meant we were at the mercy of investors for the entire life of the company. And since both we and our investors were noobs at startups, the result was a mess even by startup standards.\n",
      "\n",
      "It was a huge relief when Yahoo bought us. In principle our Viaweb stock was valuable. It was a share in a business that was profitable and growing rapidly. But it didn't feel very valuable to me; I had no idea how to value a business, but I was all too keenly aware of the near-death experiences we seemed to have every few months. Nor had I changed my grad student lifestyle significantly since we started. So when Yahoo bought us it felt like going from rags to riches. Since we were going to California, I bought a car, a yellow 1998 VW GTI. I remember thinking that its leather seats alone were by far the most luxurious thing I owned.\n",
      "\n",
      "The next year, from the summer of 1998 to the summer of 1999, must have been the least productive of my life. I didn't realize it at the time, but I was worn out from the effort and stress of running Viaweb. For a while after I got to California I tried to continue my usual m.o. of programming till 3 in the morning, but fatigue combined with Yahoo's prematurely aged culture and grim cube farm in Santa Clara gradually dragged me down. After a few months it felt disconcertingly like working at Interleaf.\n",
      "\n",
      "Yahoo had given us a lot of options when they bought us. At the time I thought Yahoo was so overvalued that they'd never be worth anything, but to my astonishment the stock went up 5x in the next year. I hung on till the first chunk of options vested, then in the summer of 1999 I left. It had been so long since I'd painted anything that I'd half forgotten why I was doing this. My brain had been entirely full of software and men's shirts for 4 years. But I had done this to get rich so I could paint, I reminded myself, and now I was rich, so I should go paint.\n",
      "\n",
      "When I said I was leaving, my boss at Yahoo had a long conversation with me about my plans. I told him all about the kinds of pictures I wanted to paint. At the time I was touched that he took such an interest ```\n",
      "```11. tried to continue my usual m.o. of programming till 3 in the morning, but fatigue combined with Yahoo's prematurely aged culture and grim cube farm in Santa Clara gradually dragged me down. After a few months it felt disconcertingly like working at Interleaf.\n",
      "\n",
      "Yahoo had given us a lot of options when they bought us. At the time I thought Yahoo was so overvalued that they'd never be worth anything, but to my astonishment the stock went up 5x in the next year. I hung on till the first chunk of options vested, then in the summer of 1999 I left. It had been so long since I'd painted anything that I'd half forgotten why I was doing this. My brain had been entirely full of software and men's shirts for 4 years. But I had done this to get rich so I could paint, I reminded myself, and now I was rich, so I should go paint.\n",
      "\n",
      "When I said I was leaving, my boss at Yahoo had a long conversation with me about my plans. I told him all about the kinds of pictures I wanted to paint. At the time I was touched that he took such an interest in me. Now I realize it was because he thought I was lying. My options at that point were worth about $2 million a month. If I was leaving that kind of money on the table, it could only be to go and start some new startup, and if I did, I might take people with me. This was the height of the Internet Bubble, and Yahoo was ground zero of it. My boss was at that moment a billionaire. Leaving then to start a new startup must have seemed to him an insanely, and yet also plausibly, ambitious plan.\n",
      "\n",
      "But I really was quitting to paint, and I started immediately. There was no time to lose. I'd already burned 4 years getting rich. Now when I talk to founders who are leaving after selling their companies, my advice is always the same: take a vacation. That's what I should have done, just gone off somewhere and done nothing for a month or two, but the idea never occurred to me.\n",
      "\n",
      "So I tried to paint, but I just didn't seem to have any energy or ambition. Part of the problem was that I didn't know many people in California. I'd compounded this problem by buying a house up in the Santa Cruz Mountains, with a beautiful view but miles from anywhere. I stuck it out for a few more months, then in desperation I went back to New York, where unless you understand about rent control you'll be surprised to hear I still had my apartment, sealed up like a tomb of my old life. Idelle was in New York at least, and there were other people trying to paint there, even though I didn't know any of them.\n",
      "\n",
      "When I got back to New York I resumed my old life, except now I was rich. It was as weird as it sounds. I resumed all my old patterns, except now there were doors where there hadn't been. Now when I was tired of walking, all I had to do was raise my hand, and (unless it was raining) a taxi would stop to pick me up. Now when I walked past charming little restaurants I could go in and order lunch. It was exciting for a while. Painting started to go better. I experimented with a new kind of still life where I'd paint one painting in the old way, then photograph it and print it, blown up, on canvas, and then use that as the underpainting for a second still life, painted from the same objects (which hopefully hadn't rotted yet).\n",
      "\n",
      "Meanwhile I looked for an apartment to buy. Now I could actually choose what neighborhood to live in. Where, I asked myself and various real estate agents, is the Cambridge of New York? Aided by occasional visits to actual Cambridge, I gradually realized there wasn't one. Huh.\n",
      "\n",
      "Around this time, in the spring of 2000, I had an idea. It was clear from our experience with Viaweb that web apps were the future. Why not build a web app for making web apps? Why not let people edit code on our server through the browser, and then host the resulting applications for them? [9] You could run all sorts of services on the servers that these applications could use just by making an API call: making and receiving phone calls, manipulating images, taking credit card payments, etc.\n",
      "\n",
      "I got so excited about this idea that I couldn't think about anything else. It seemed obvious that this was the future. I didn't particularly want to start another company, but it was clear that ```\n",
      "```12. photograph it and print it, blown up, on canvas, and then use that as the underpainting for a second still life, painted from the same objects (which hopefully hadn't rotted yet).\n",
      "\n",
      "Meanwhile I looked for an apartment to buy. Now I could actually choose what neighborhood to live in. Where, I asked myself and various real estate agents, is the Cambridge of New York? Aided by occasional visits to actual Cambridge, I gradually realized there wasn't one. Huh.\n",
      "\n",
      "Around this time, in the spring of 2000, I had an idea. It was clear from our experience with Viaweb that web apps were the future. Why not build a web app for making web apps? Why not let people edit code on our server through the browser, and then host the resulting applications for them? [9] You could run all sorts of services on the servers that these applications could use just by making an API call: making and receiving phone calls, manipulating images, taking credit card payments, etc.\n",
      "\n",
      "I got so excited about this idea that I couldn't think about anything else. It seemed obvious that this was the future. I didn't particularly want to start another company, but it was clear that this idea would have to be embodied as one, so I decided to move to Cambridge and start it. I hoped to lure Robert into working on it with me, but there I ran into a hitch. Robert was now a postdoc at MIT, and though he'd made a lot of money the last time I'd lured him into working on one of my schemes, it had also been a huge time sink. So while he agreed that it sounded like a plausible idea, he firmly refused to work on it.\n",
      "\n",
      "Hmph. Well, I'd do it myself then. I recruited Dan Giffin, who had worked for Viaweb, and two undergrads who wanted summer jobs, and we got to work trying to build what it's now clear is about twenty companies and several open source projects worth of software. The language for defining applications would of course be a dialect of Lisp. But I wasn't so naive as to assume I could spring an overt Lisp on a general audience; we'd hide the parentheses, like Dylan did.\n",
      "\n",
      "By then there was a name for the kind of company Viaweb was, an \"application service provider,\" or ASP. This name didn't last long before it was replaced by \"software as a service,\" but it was current for long enough that I named this new company after it: it was going to be called Aspra.\n",
      "\n",
      "I started working on the application builder, Dan worked on network infrastructure, and the two undergrads worked on the first two services (images and phone calls). But about halfway through the summer I realized I really didn't want to run a company — especially not a big one, which it was looking like this would have to be. I'd only started Viaweb because I needed the money. Now that I didn't need money anymore, why was I doing this? If this vision had to be realized as a company, then screw the vision. I'd build a subset that could be done as an open source project.\n",
      "\n",
      "Much to my surprise, the time I spent working on this stuff was not wasted after all. After we started Y Combinator, I would often encounter startups working on parts of this new architecture, and it was very useful to have spent so much time thinking about it and even trying to write some of it.\n",
      "\n",
      "The subset I would build as an open source project was the new Lisp, whose parentheses I now wouldn't even have to hide. A lot of Lisp hackers dream of building a new Lisp, partly because one of the distinctive features of the language is that it has dialects, and partly, I think, because we have in our minds a Platonic form of Lisp that all existing dialects fall short of. I certainly did. So at the end of the summer Dan and I switched to working on this new dialect of Lisp, which I called Arc, in a house I bought in Cambridge.\n",
      "\n",
      "The following spring, lightning struck. I was invited to give a talk at a Lisp conference, so I gave one about how we'd used Lisp at Viaweb. Afterward I put a postscript file of this talk online, on paulgraham.com, which I'd created years before using Viaweb but had never used for anything. In one day it got 30,000 page views. What on earth had happened? The referring urls showed that someone had posted it on Slashdot. [10]\n",
      "\n",
      "Wow, I thought, there's an audience. If I write something and put it on the web, anyone can read it. That may seem obvious now, but it was surprising then. In the print ```\n",
      "```13. new Lisp, whose parentheses I now wouldn't even have to hide. A lot of Lisp hackers dream of building a new Lisp, partly because one of the distinctive features of the language is that it has dialects, and partly, I think, because we have in our minds a Platonic form of Lisp that all existing dialects fall short of. I certainly did. So at the end of the summer Dan and I switched to working on this new dialect of Lisp, which I called Arc, in a house I bought in Cambridge.\n",
      "\n",
      "The following spring, lightning struck. I was invited to give a talk at a Lisp conference, so I gave one about how we'd used Lisp at Viaweb. Afterward I put a postscript file of this talk online, on paulgraham.com, which I'd created years before using Viaweb but had never used for anything. In one day it got 30,000 page views. What on earth had happened? The referring urls showed that someone had posted it on Slashdot. [10]\n",
      "\n",
      "Wow, I thought, there's an audience. If I write something and put it on the web, anyone can read it. That may seem obvious now, but it was surprising then. In the print era there was a narrow channel to readers, guarded by fierce monsters known as editors. The only way to get an audience for anything you wrote was to get it published as a book, or in a newspaper or magazine. Now anyone could publish anything.\n",
      "\n",
      "This had been possible in principle since 1993, but not many people had realized it yet. I had been intimately involved with building the infrastructure of the web for most of that time, and a writer as well, and it had taken me 8 years to realize it. Even then it took me several years to understand the implications. It meant there would be a whole new generation of essays. [11]\n",
      "\n",
      "In the print era, the channel for publishing essays had been vanishingly small. Except for a few officially anointed thinkers who went to the right parties in New York, the only people allowed to publish essays were specialists writing about their specialties. There were so many essays that had never been written, because there had been no way to publish them. Now they could be, and I was going to write them. [12]\n",
      "\n",
      "I've worked on several different things, but to the extent there was a turning point where I figured out what to work on, it was when I started publishing essays online. From then on I knew that whatever else I did, I'd always write essays too.\n",
      "\n",
      "I knew that online essays would be a marginal medium at first. Socially they'd seem more like rants posted by nutjobs on their GeoCities sites than the genteel and beautifully typeset compositions published in The New Yorker. But by this point I knew enough to find that encouraging instead of discouraging.\n",
      "\n",
      "One of the most conspicuous patterns I've noticed in my life is how well it has worked, for me at least, to work on things that weren't prestigious. Still life has always been the least prestigious form of painting. Viaweb and Y Combinator both seemed lame when we started them. I still get the glassy eye from strangers when they ask what I'm writing, and I explain that it's an essay I'm going to publish on my web site. Even Lisp, though prestigious intellectually in something like the way Latin is, also seems about as hip.\n",
      "\n",
      "It's not that unprestigious types of work are good per se. But when you find yourself drawn to some kind of work despite its current lack of prestige, it's a sign both that there's something real to be discovered there, and that you have the right kind of motives. Impure motives are a big danger for the ambitious. If anything is going to lead you astray, it will be the desire to impress people. So while working on things that aren't prestigious doesn't guarantee you're on the right track, it at least guarantees you're not on the most common type of wrong one.\n",
      "\n",
      "Over the next several years I wrote lots of essays about all kinds of different topics. O'Reilly reprinted a collection of them as a book, called Hackers & Painters after one of the essays in it. I also worked on spam filters, and did some more painting. I used to have dinners for a group of friends every thursday night, which taught me how to cook for groups. And I bought another building in Cambridge, a former candy factory (and later, twas said, porn studio), to use as an office.\n",
      "\n",
      "One night in October 2003 there was a big party at my house. It was a clever idea of my friend Maria Daniels, who was one of the thursday diners. Three separate hosts ```\n",
      "```14. current lack of prestige, it's a sign both that there's something real to be discovered there, and that you have the right kind of motives. Impure motives are a big danger for the ambitious. If anything is going to lead you astray, it will be the desire to impress people. So while working on things that aren't prestigious doesn't guarantee you're on the right track, it at least guarantees you're not on the most common type of wrong one.\n",
      "\n",
      "Over the next several years I wrote lots of essays about all kinds of different topics. O'Reilly reprinted a collection of them as a book, called Hackers & Painters after one of the essays in it. I also worked on spam filters, and did some more painting. I used to have dinners for a group of friends every thursday night, which taught me how to cook for groups. And I bought another building in Cambridge, a former candy factory (and later, twas said, porn studio), to use as an office.\n",
      "\n",
      "One night in October 2003 there was a big party at my house. It was a clever idea of my friend Maria Daniels, who was one of the thursday diners. Three separate hosts would all invite their friends to one party. So for every guest, two thirds of the other guests would be people they didn't know but would probably like. One of the guests was someone I didn't know but would turn out to like a lot: a woman called Jessica Livingston. A couple days later I asked her out.\n",
      "\n",
      "Jessica was in charge of marketing at a Boston investment bank. This bank thought it understood startups, but over the next year, as she met friends of mine from the startup world, she was surprised how different reality was. And how colorful their stories were. So she decided to compile a book of interviews with startup founders.\n",
      "\n",
      "When the bank had financial problems and she had to fire half her staff, she started looking for a new job. In early 2005 she interviewed for a marketing job at a Boston VC firm. It took them weeks to make up their minds, and during this time I started telling her about all the things that needed to be fixed about venture capital. They should make a larger number of smaller investments instead of a handful of giant ones, they should be funding younger, more technical founders instead of MBAs, they should let the founders remain as CEO, and so on.\n",
      "\n",
      "One of my tricks for writing essays had always been to give talks. The prospect of having to stand up in front of a group of people and tell them something that won't waste their time is a great spur to the imagination. When the Harvard Computer Society, the undergrad computer club, asked me to give a talk, I decided I would tell them how to start a startup. Maybe they'd be able to avoid the worst of the mistakes we'd made.\n",
      "\n",
      "So I gave this talk, in the course of which I told them that the best sources of seed funding were successful startup founders, because then they'd be sources of advice too. Whereupon it seemed they were all looking expectantly at me. Horrified at the prospect of having my inbox flooded by business plans (if I'd only known), I blurted out \"But not me!\" and went on with the talk. But afterward it occurred to me that I should really stop procrastinating about angel investing. I'd been meaning to since Yahoo bought us, and now it was 7 years later and I still hadn't done one angel investment.\n",
      "\n",
      "Meanwhile I had been scheming with Robert and Trevor about projects we could work on together. I missed working with them, and it seemed like there had to be something we could collaborate on.\n",
      "\n",
      "As Jessica and I were walking home from dinner on March 11, at the corner of Garden and Walker streets, these three threads converged. Screw the VCs who were taking so long to make up their minds. We'd start our own investment firm and actually implement the ideas we'd been talking about. I'd fund it, and Jessica could quit her job and work for it, and we'd get Robert and Trevor as partners too. [13]\n",
      "\n",
      "Once again, ignorance worked in our favor. We had no idea how to be angel investors, and in Boston in 2005 there were no Ron Conways to learn from. So we just made what seemed like the obvious choices, and some of the things we did turned out to be novel.\n",
      "\n",
      "There are multiple components to Y Combinator, and we didn't figure them all out at once. The part we got first was to be an angel firm. In those days, those two words didn't go together. There were VC firms, which ```\n",
      "```15. one angel investment.\n",
      "\n",
      "Meanwhile I had been scheming with Robert and Trevor about projects we could work on together. I missed working with them, and it seemed like there had to be something we could collaborate on.\n",
      "\n",
      "As Jessica and I were walking home from dinner on March 11, at the corner of Garden and Walker streets, these three threads converged. Screw the VCs who were taking so long to make up their minds. We'd start our own investment firm and actually implement the ideas we'd been talking about. I'd fund it, and Jessica could quit her job and work for it, and we'd get Robert and Trevor as partners too. [13]\n",
      "\n",
      "Once again, ignorance worked in our favor. We had no idea how to be angel investors, and in Boston in 2005 there were no Ron Conways to learn from. So we just made what seemed like the obvious choices, and some of the things we did turned out to be novel.\n",
      "\n",
      "There are multiple components to Y Combinator, and we didn't figure them all out at once. The part we got first was to be an angel firm. In those days, those two words didn't go together. There were VC firms, which were organized companies with people whose job it was to make investments, but they only did big, million dollar investments. And there were angels, who did smaller investments, but these were individuals who were usually focused on other things and made investments on the side. And neither of them helped founders enough in the beginning. We knew how helpless founders were in some respects, because we remembered how helpless we'd been. For example, one thing Julian had done for us that seemed to us like magic was to get us set up as a company. We were fine writing fairly difficult software, but actually getting incorporated, with bylaws and stock and all that stuff, how on earth did you do that? Our plan was not only to make seed investments, but to do for startups everything Julian had done for us.\n",
      "\n",
      "YC was not organized as a fund. It was cheap enough to run that we funded it with our own money. That went right by 99% of readers, but professional investors are thinking \"Wow, that means they got all the returns.\" But once again, this was not due to any particular insight on our part. We didn't know how VC firms were organized. It never occurred to us to try to raise a fund, and if it had, we wouldn't have known where to start. [14]\n",
      "\n",
      "The most distinctive thing about YC is the batch model: to fund a bunch of startups all at once, twice a year, and then to spend three months focusing intensively on trying to help them. That part we discovered by accident, not merely implicitly but explicitly due to our ignorance about investing. We needed to get experience as investors. What better way, we thought, than to fund a whole bunch of startups at once? We knew undergrads got temporary jobs at tech companies during the summer. Why not organize a summer program where they'd start startups instead? We wouldn't feel guilty for being in a sense fake investors, because they would in a similar sense be fake founders. So while we probably wouldn't make much money out of it, we'd at least get to practice being investors on them, and they for their part would probably have a more interesting summer than they would working at Microsoft.\n",
      "\n",
      "We'd use the building I owned in Cambridge as our headquarters. We'd all have dinner there once a week — on tuesdays, since I was already cooking for the thursday diners on thursdays — and after dinner we'd bring in experts on startups to give talks.\n",
      "\n",
      "We knew undergrads were deciding then about summer jobs, so in a matter of days we cooked up something we called the Summer Founders Program, and I posted an announcement on my site, inviting undergrads to apply. I had never imagined that writing essays would be a way to get \"deal flow,\" as investors call it, but it turned out to be the perfect source. [15] We got 225 applications for the Summer Founders Program, and we were surprised to find that a lot of them were from people who'd already graduated, or were about to that spring. Already this SFP thing was starting to feel more serious than we'd intended.\n",
      "\n",
      "We invited about 20 of the 225 groups to interview in person, and from those we picked 8 to fund. They were an impressive group. That first batch included reddit, Justin Kan and Emmett Shear, who went on to found Twitch, Aaron Swartz, who had already helped write the RSS spec and would a few years later become a martyr for open access, and ```\n",
      "</Documents>\n",
      "</SYS>\n",
      "Query: None\n",
      "You:\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from lightrag.components.retriever import LLMRetriever\n",
    "from lightrag.components.model_client import OpenAIClient\n",
    "\n",
    "from lightrag.tracing import trace_generator_call\n",
    "\n",
    "from lightrag.utils import setup_env\n",
    "\n",
    "# 1. set up the tracing for failed call as the retriever has generator attribute\n",
    "\n",
    "@trace_generator_call(save_dir=\"developer_notes/traces\")\n",
    "class LoggedLLMRetriever(LLMRetriever):\n",
    "    pass\n",
    "top_k = 2\n",
    "retriever = LoggedLLMRetriever(\n",
    "    top_k = top_k, model_client=OpenAIClient(), model_kwargs={\"model\": \"gpt-3.5-turbo\"}\n",
    ")\n",
    "\n",
    "retriever.build_index_from_documents(documents=[doc.text for doc in splitted_documents[0:16]])\n",
    "\n",
    "print(retriever)\n",
    "retriever.generator.print_prompt()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note: We need to know the ground truth, you can save the splitted documents and then label the data.\n",
    "\n",
    "Here we did that, the ground truth is (indices)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[RetrieverOutput(doc_indices=[1], doc_scores=None, query='What happened at Viaweb and Interleaf?', documents=None)]\n"
     ]
    }
   ],
   "source": [
    "query = \"What happened at Viaweb and Interleaf?\"\n",
    "output = retriever(input=query)\n",
    "print(output)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "output.documents [Document(id=619bb230-3a9b-4258-b7c9-8846aea0db55, text=was that the other fields took up so much of the space of ideas that there wasn't much left for these supposed ultimate truths. All that seemed left for philosophy were edge cases that people in other fields felt could safely be ignored.\n",
      "\n",
      "I couldn't have put this into words when I was 18. All I knew at the time was that I kept taking philosophy courses and they kept being boring. So I decided to s ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=958, parent_doc_id=4e32f7b4-a82d-415b-a1e8-e63ef07e9ac6)]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "1"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# output[0].documents = [splitted_documents[idx] for idx in output[0].doc_indices]\n",
    "for per_query_output in output:\n",
    "    per_query_output.documents = [splitted_documents[idx] for idx in per_query_output.doc_indices]\n",
    "print(\"output.documents\", output[0].documents)\n",
    "len(output)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "was that the other fields took up so much of the space of ideas that there wasn't much left for these supposed ultimate truths. All that seemed left for philosophy were edge cases that people in other fields felt could safely be ignored.\n",
      "\n",
      "I couldn't have put this into words when I was 18. All I knew at the time was that I kept taking philosophy courses and they kept being boring. So I decided to switch to AI.\n",
      "\n",
      "AI was in the air in the mid 1980s, but there were two things especially that made me want to work on it: a novel by Heinlein called The Moon is a Harsh Mistress, which featured an intelligent computer called Mike, and a PBS documentary that showed Terry Winograd using SHRDLU. I haven't tried rereading The Moon is a Harsh Mistress, so I don't know how well it has aged, but when I read it I was drawn entirely into its world. It seemed only a matter of time before we'd have Mike, and when I saw Winograd using SHRDLU, it seemed like that time would be a few years at most. All you had to do was teach SHRDLU more words.\n",
      "\n",
      "There weren't any classes in AI at Cornell then, not even graduate classes, so I started trying to teach myself. Which meant learning Lisp, since in those days Lisp was regarded as the language of AI. The commonly used programming languages then were pretty primitive, and programmers' ideas correspondingly so. The default language at Cornell was a Pascal-like language called PL/I, and the situation was similar elsewhere. Learning Lisp expanded my concept of a program so fast that it was years before I started to have a sense of where the new limits were. This was more like it; this was what I had expected college to do. It wasn't happening in a class, like it was supposed to, but that was ok. For the next couple years I was on a roll. I knew what I was going to do.\n",
      "\n",
      "For my undergraduate thesis, I reverse-engineered SHRDLU. My God did I love working on that program. It was a pleasing bit of code, but what made it even more exciting was my belief — hard to imagine now, but not unique in 1985 — that it was already climbing the lower slopes of intelligence.\n",
      "\n",
      "I had gotten into a program at Cornell that didn't make you choose a major. You could take whatever classes you liked, and choose whatever you liked to put on your degree. I of course chose \"Artificial Intelligence.\" When I got the actual physical diploma, I was dismayed to find that the quotes had been included, which made them read as scare-quotes. At the time this bothered me, but now it seems amusingly accurate, for reasons I was about to discover.\n",
      "\n",
      "I applied to 3 grad schools: MIT and Yale, which were renowned for AI at the time, and Harvard, which I'd visited because Rich Draves went there, and was also home to Bill Woods, who'd invented the type of parser I used in my SHRDLU clone. Only Harvard accepted me, so that was where I went.\n",
      "\n",
      "I don't remember the moment it happened, or if there even was a specific moment, but during the first year of grad school I realized that AI, as practiced at the time, was a hoax. By which I mean the sort of AI in which a program that's told \"the dog is sitting on the chair\" translates this into some formal representation and adds it to the list of things it knows.\n",
      "\n",
      "What these programs really showed was that there's a subset of natural language that's a formal language. But a very proper subset. It was clear that there was an unbridgeable gap between what they could do and actually understanding natural language. It was not, in fact, simply a matter of teaching SHRDLU more words. That whole way of doing AI, with explicit data structures representing concepts, was not going to work. Its brokenness did, as so often happens, generate a lot of opportunities to write papers about various band-aids that could be applied to it, but it was never going to get us Mike.\n",
      "\n",
      "So I looked around to see what I could salvage from the wreckage of my plans, and there was Lisp. I knew from experience that Lisp was interesting for its own sake and not just for its association with AI, even though that was the main reason people cared about it at the time. So I decided to focus on Lisp. In fact, I decided to write a book about Lisp hacking. It's scary to think how little I knew about Lisp hacking when I started writing that book. But there's nothing like writing a book about something to help \n",
      "False\n",
      "False\n"
     ]
    }
   ],
   "source": [
    "# check the first document\n",
    "print(output[0].documents[0].text)\n",
    "print(\"interleaf\" in output[0].documents[0].text.lower())\n",
    "print(\"viaweb\" in output[0].documents[0].text.lower())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "ename": "IndexError",
     "evalue": "list index out of range",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mIndexError\u001b[0m                                Traceback (most recent call last)",
      "Cell \u001b[0;32mIn[27], line 2\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[38;5;66;03m# check the second document\u001b[39;00m\n\u001b[0;32m----> 2\u001b[0m \u001b[38;5;28mprint\u001b[39m(\u001b[43moutput\u001b[49m\u001b[43m[\u001b[49m\u001b[38;5;241;43m0\u001b[39;49m\u001b[43m]\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mdocuments\u001b[49m\u001b[43m[\u001b[49m\u001b[38;5;241;43m1\u001b[39;49m\u001b[43m]\u001b[49m\u001b[38;5;241m.\u001b[39mtext)\n\u001b[1;32m      3\u001b[0m \u001b[38;5;28mprint\u001b[39m(\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124minterleaf\u001b[39m\u001b[38;5;124m\"\u001b[39m \u001b[38;5;129;01min\u001b[39;00m output[\u001b[38;5;241m0\u001b[39m]\u001b[38;5;241m.\u001b[39mdocuments[\u001b[38;5;241m1\u001b[39m]\u001b[38;5;241m.\u001b[39mtext\u001b[38;5;241m.\u001b[39mlower())\n\u001b[1;32m      4\u001b[0m \u001b[38;5;28mprint\u001b[39m(\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mviaweb\u001b[39m\u001b[38;5;124m\"\u001b[39m \u001b[38;5;129;01min\u001b[39;00m output[\u001b[38;5;241m0\u001b[39m]\u001b[38;5;241m.\u001b[39mdocuments[\u001b[38;5;241m1\u001b[39m]\u001b[38;5;241m.\u001b[39mtext\u001b[38;5;241m.\u001b[39mlower())\n",
      "\u001b[0;31mIndexError\u001b[0m: list index out of range"
     ]
    }
   ],
   "source": [
    "# check the second document\n",
    "print(output[0].documents[1].text)\n",
    "print(\"interleaf\" in output[0].documents[1].text.lower())\n",
    "print(\"viaweb\" in output[0].documents[1].text.lower())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Reranker\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# from lightrag.components.retriever import RerankerRetriever\n",
    "\n",
    "# query = \"Li\"\n",
    "# strings = [\"Li\", \"text2\"]\n",
    "\n",
    "# retriever = RerankerRetriever(top_k=1)\n",
    "# print(retriever)\n",
    "# retriever.build_index_from_documents(documents=documents)\n",
    "# print(retriever.documents)\n",
    "# output = retriever.retrieve(query)\n",
    "# print(output)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# retriever.build_index_from_documents(documents=strings)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# output = retriever.retrieve(query)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## FAISSRetriever\n",
    "\n",
    "To use Semantic search, we very likely need TextSplitter and compute the embeddings. This data-preprocessing is more use-case specific and should be better to be done by users in data transformation stage. Then we can treat these embeddings as the input documents.\n",
    "\n",
    "In this case, the real index is the splitted documents along with its embeddings. We will use ``LocalDB`` to handle the data transformation and the storage of the index.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from lightrag.core.db import LocalDB\n",
    "\n",
    "db = LocalDB()\n",
    "db.load_documents(documents)\n",
    "len(db.documents)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us see how to create data transformers using only the component config"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create data transformer\n",
    "data_transformer_config = {  # attribute and its config to recreate the component\n",
    "        \"embedder\":{\n",
    "            \"component_name\": \"Embedder\",\n",
    "            \"component_config\": {\n",
    "                \"model_client\": {\n",
    "                    \"component_name\": \"OpenAIClient\",\n",
    "                    \"component_config\": {},\n",
    "                },\n",
    "                \"model_kwargs\": {\n",
    "                    \"model\": \"text-embedding-3-small\",\n",
    "                    \"dimensions\": 256,\n",
    "                    \"encoding_format\": \"float\",\n",
    "                },\n",
    "            },\n",
    "        },\n",
    "        \"document_splitter\": {\n",
    "            \"component_name\": \"DocumentSplitter\",\n",
    "            \"component_config\": {\n",
    "                \"split_by\": \"word\",\n",
    "                \"split_length\": 400,\n",
    "                \"split_overlap\": 200,\n",
    "            },\n",
    "        },\n",
    "        \"to_embeddings\": {\n",
    "            \"component_name\": \"ToEmbeddings\",\n",
    "            \"component_config\": {\n",
    "                \"vectorizer\": {\n",
    "                    \"component_name\": \"Embedder\",\n",
    "                    \"component_config\": {\n",
    "                        \"model_client\": {\n",
    "                            \"component_name\": \"OpenAIClient\",\n",
    "                            \"component_config\": {},\n",
    "                        },\n",
    "                        \"model_kwargs\": {\n",
    "                            \"model\": \"text-embedding-3-small\",\n",
    "                            \"dimensions\": 256,\n",
    "                            \"encoding_format\": \"float\",\n",
    "                        },\n",
    "                    },\n",
    "                    # the other config is to instantiate the entity (class and function) with the given config as arguments\n",
    "                    # \"entity_state\": \"storage/embedder.pkl\", # this will load back the state of the entity\n",
    "                },\n",
    "                \"batch_size\": 100,\n",
    "            },\n",
    "        },\n",
    "    }"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'embedder': Embedder(\n",
      "  model_kwargs={'model': 'text-embedding-3-small', 'dimensions': 256, 'encoding_format': 'float'}, \n",
      "  (model_client): OpenAIClient()\n",
      "), 'document_splitter': DocumentSplitter(split_by=word, split_length=400, split_overlap=200), 'to_embeddings': ToEmbeddings(\n",
      "  batch_size=100\n",
      "  (vectorizer): Embedder(\n",
      "    model_kwargs={'model': 'text-embedding-3-small', 'dimensions': 256, 'encoding_format': 'float'}, \n",
      "    (model_client): OpenAIClient()\n",
      "  )\n",
      "  (batch_embedder): BatchEmbedder(\n",
      "    (embedder): Embedder(\n",
      "      model_kwargs={'model': 'text-embedding-3-small', 'dimensions': 256, 'encoding_format': 'float'}, \n",
      "      (model_client): OpenAIClient()\n",
      "    )\n",
      "  )\n",
      ")}\n"
     ]
    }
   ],
   "source": [
    "from lightrag.utils.config import new_components_from_config\n",
    "\n",
    "components = new_components_from_config(data_transformer_config)\n",
    "print(components)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Sequential(\n",
       "  (0): DocumentSplitter(split_by=word, split_length=400, split_overlap=200)\n",
       "  (1): ToEmbeddings(\n",
       "    batch_size=100\n",
       "    (vectorizer): Embedder(\n",
       "      model_kwargs={'model': 'text-embedding-3-small', 'dimensions': 256, 'encoding_format': 'float'}, \n",
       "      (model_client): OpenAIClient()\n",
       "    )\n",
       "    (batch_embedder): BatchEmbedder(\n",
       "      (embedder): Embedder(\n",
       "        model_kwargs={'model': 'text-embedding-3-small', 'dimensions': 256, 'encoding_format': 'float'}, \n",
       "        (model_client): OpenAIClient()\n",
       "      )\n",
       "    )\n",
       "  )\n",
       ")"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from lightrag.core.component import Sequential\n",
    "\n",
    "data_transformer = Sequential(components[\"document_splitter\"], components[\"to_embeddings\"])\n",
    "data_transformer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The above code is equivalent to the code with config\n",
    "\n",
    "```python\n",
    "\n",
    "        vectorizer = Embedder(\n",
    "            model_client=OpenAIClient(),\n",
    "            # batch_size=self.vectorizer_settings[\"batch_size\"],\n",
    "            \n",
    "            model_kwargs=self.vectorizer_settings[\"model_kwargs\"],\n",
    "        )\n",
    "        # TODO: check document splitter, how to process the parent and order of the chunks\n",
    "        text_splitter = DocumentSplitter(\n",
    "            split_by=self.text_splitter_settings[\"split_by\"],\n",
    "            split_length=self.text_splitter_settings[\"chunk_size\"],\n",
    "            split_overlap=self.text_splitter_settings[\"chunk_overlap\"],\n",
    "        )\n",
    "        self.data_transformer = Sequential(\n",
    "            text_splitter,\n",
    "            ToEmbeddings(\n",
    "                vectorizer=vectorizer,\n",
    "                batch_size=self.vectorizer_settings[\"batch_size\"],\n",
    "            ),\n",
    "        )\n",
    "```\n",
    "\n",
    "Config:\n",
    "\n",
    "```yaml\n",
    "vectorizer:\n",
    "  batch_size: 100\n",
    "  model_kwargs:\n",
    "    model: text-embedding-3-small\n",
    "    dimensions: 256\n",
    "    encoding_format: float\n",
    "\n",
    "retriever:\n",
    "  top_k: 2\n",
    "\n",
    "generator:\n",
    "  model: gpt-3.5-turbo\n",
    "  temperature: 0.3\n",
    "  stream: false\n",
    "\n",
    "text_splitter:\n",
    "  split_by: word\n",
    "  chunk_size: 400\n",
    "  chunk_overlap: 200\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Splitting documents: 100%|██████████| 1/1 [00:00<00:00,  8.55it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[Document(id=2a1cc40e-4172-471d-b741-d44af5548277, text=\n",
      "\n",
      "What I Worked On\n",
      "\n",
      "February 2021\n",
      "\n",
      "Before college the two main things I worked on, outside of school, were writing and programming. I didn't write essays. I wrote what beginning writers were supposed to write then, and probably still are: short stories. My stories were awful. They had hardly any plot, just characters with strong feelings, which I imagined made them deep.\n",
      "\n",
      "The first programs I trie ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=499, parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=3923c220-13ba-4886-a363-18320a006e6d, text=spectacularly loud printer.\n",
      "\n",
      "I was puzzled by the 1401. I couldn't figure out what to do with it. And in retrospect there's not much I could have done with it. The only form of input to programs was data stored on punched cards, and I didn't have any data stored on punched cards. The only other option was to do things that didn't rely on any input, like calculate approximations of pi, but I didn't ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=493, parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=792c5a32-2a64-40a5-b356-e1312327e7c1, text=was sold as a kit by Heathkit. I remember vividly how impressed and envious I felt watching him sitting in front of it, typing programs right into the computer.\n",
      "\n",
      "Computers were expensive in those days and it took me years of nagging before I convinced my father to buy one, a TRS-80, in about 1980. The gold standard then was the Apple II, but a TRS-80 was good enough. This was when I really started ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=487, parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=c4554a05-4391-433c-a267-0c63cd086150, text=was that the other fields took up so much of the space of ideas that there wasn't much left for these supposed ultimate truths. All that seemed left for philosophy were edge cases that people in other fields felt could safely be ignored.\n",
      "\n",
      "I couldn't have put this into words when I was 18. All I knew at the time was that I kept taking philosophy courses and they kept being boring. So I decided to s ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=480, parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=fb02bba2-aec8-4a98-be7e-b30c0fe91fa3, text=any classes in AI at Cornell then, not even graduate classes, so I started trying to teach myself. Which meant learning Lisp, since in those days Lisp was regarded as the language of AI. The commonly used programming languages then were pretty primitive, and programmers' ideas correspondingly so. The default language at Cornell was a Pascal-like language called PL/I, and the situation was similar  ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=482, parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=920f1e00-0e70-4d77-a8bd-a17fa9b5f982, text=didn't make you choose a major. You could take whatever classes you liked, and choose whatever you liked to put on your degree. I of course chose \"Artificial Intelligence.\" When I got the actual physical diploma, I was dismayed to find that the quotes had been included, which made them read as scare-quotes. At the time this bothered me, but now it seems amusingly accurate, for reasons I was about  ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=479, parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=a3243068-9bfd-4f6d-a8e8-1981d18cd6db, text=really showed was that there's a subset of natural language that's a formal language. But a very proper subset. It was clear that there was an unbridgeable gap between what they could do and actually understanding natural language. It was not, in fact, simply a matter of teaching SHRDLU more words. That whole way of doing AI, with explicit data structures representing concepts, was not going to wo ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=479, parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=ffb4abe2-8328-401e-bf49-4c083fce8d53, text=you learn it. The book, On Lisp, wasn't published till 1993, but I wrote much of it in grad school.\n",
      "\n",
      "Computer Science is an uneasy alliance between two halves, theory and systems. The theory people prove things, and the systems people build things. I wanted to build things. I had plenty of respect for theory — indeed, a sneaking suspicion that it was the more admirable of the two halves — but buil ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=488, parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=7452f006-bbab-43da-b3cb-3d17a8aaa557, text=That was what happened to systems work.\n",
      "\n",
      "I wanted not just to build things, but to build things that would last.\n",
      "\n",
      "In this dissatisfied state I went in 1988 to visit Rich Draves at CMU, where he was in grad school. One day I went to visit the Carnegie Institute, where I'd spent a lot of time as a kid. While looking at a painting there I realized something that might seem obvious, but was a big surp ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=480, parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=0a67c2e9-85e5-4a59-af75-2adc0efb6fbc, text=didn't just appear spontaneously — but it was as if the people who made it were a different species. They either lived long ago or were mysterious geniuses doing strange things in profiles in Life magazine. The idea of actually being able to make art, to put that verb before that noun, seemed almost miraculous.\n",
      "\n",
      "That fall I started taking art classes at Harvard. Grad students could take classes in ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=487, parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=1985def9-d07b-4ba8-a6f0-cca4bb455589, text=to get out of grad school.\n",
      "\n",
      "Then one day in April 1990 a crack appeared in the wall. I ran into professor Cheatham and he asked if I was far enough along to graduate that June. I didn't have a word of my dissertation written, but in what must have been the quickest bit of thinking in my life, I decided to take a shot at writing one in the 5 weeks or so that remained before the deadline, reusing pa ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=497, parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=5b3eb253-e061-42d9-a5cc-244b25108943, text=the Accademia, so off to Providence I went.\n",
      "\n",
      "I'd applied for the BFA program at RISD, which meant in effect that I had to go to college again. This was not as strange as it sounds, because I was only 25, and art schools are full of people of different ages. RISD counted me as a transfer sophomore and said I had to do the foundation that summer. The foundation means the classes that everyone has to ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=488, parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=7ef5e6d8-d19c-4e97-856d-74fc4be8bc2d, text=so many stranieri attracted by the idea of studying art in Florence that the Italian students would otherwise have been outnumbered. I was in decent shape at painting and drawing from the RISD foundation that summer, but I still don't know how I managed to pass the written exam. I remember that I answered the essay question by writing about Cezanne, and that I cranked up the intellectual level as  ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=480, parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=dc047964-cc73-4c41-8dd1-251684a8540b, text=that you see in 19th century studio paintings, and a nude model sitting as close to it as possible without getting burned. Except hardly anyone else painted her besides me. The rest of the students spent their time chatting or occasionally trying to imitate things they'd seen in American art magazines.\n",
      "\n",
      "Our model turned out to live just down the street from me. She made a living from a combination ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=484, parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=9fb67b6c-d7ba-4c15-b8ac-ab132e76d4be, text=generic person, which you then modify to match the specific person you're painting. Whereas a still life you can, if you want, copy pixel by pixel from what you're seeing. You don't want to stop there, of course, or you get merely photographic accuracy, and what makes a still life interesting is that it's been through a head. You want to emphasize the visual cues that tell you, for example, that t ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=486, parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=90ee2b5b-86c7-43b5-8fee-424fb982e86e, text=to notice every leaf on every bush. But when you have to paint something, you have to look more closely, and when you do there's a lot to see. You can still be noticing new things after days of trying to paint something people usually take for granted, just as you can after days of trying to write an essay about something people usually take for granted.\n",
      "\n",
      "This is not the only way to paint. I'm not ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=479, parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=95c90d31-99d6-49aa-9b62-3c7f304949ed, text=software for creating documents. You mean like Microsoft Word? Exactly. That was how I learned that low end software tends to eat high end software. But Interleaf still had a few years to live yet. [5]\n",
      "\n",
      "Interleaf had done something pretty bold. Inspired by Emacs, they'd added a scripting language, and even made the scripting language a dialect of Lisp. Now they wanted a Lisp hacker to write things ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=469, parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=282c1e7e-448d-49cc-bf03-817e04b61bda, text=this time gotten a contract to publish.\n",
      "\n",
      "The good part was that I got paid huge amounts of money, especially by art student standards. In Florence, after paying my part of the rent, my budget for everything else had been $7 a day. Now I was getting paid more than 4 times that every hour, even when I was just sitting in a meeting. By living cheaply I not only managed to save enough to go back to RI ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=479, parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=32880b8e-7792-4fbb-9011-fc3d7e3d360b, text=it.\n",
      "\n",
      "But the most important thing I learned, and which I used in both Viaweb and Y Combinator, is that the low end eats the high end: that it's good to be the \"entry level\" option, even though that will be less prestigious, because if you're not, someone else will be, and will squash you against the ceiling. Which in turn means that prestige is a danger sign.\n",
      "\n",
      "When I left to go back to RISD the ne ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=489, parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=56406cc7-944f-43d8-88e0-4c87f21b1e80, text=it was more like the Accademia than not. Better organized, certainly, and a lot more expensive, but it was now becoming clear that art school did not bear the same relationship to art that medical school bore to medicine. At least not the painting department. The textile department, which my next door neighbor belonged to, seemed to be pretty rigorous. No doubt illustration and architecture were t ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=484, parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=5575ca2b-73b9-4645-a7e3-6a021301a9eb, text=kids who \"could draw\" in high school, and now had come to what was supposed to be the best art school in the country, to learn to draw even better. They tended to be confused and demoralized by what they found at RISD, but they kept going, because painting was what they did. I was not one of the kids who could draw in high school, but at RISD I was definitely closer to their tribe than the tribe o ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=485, parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=9e729645-afca-4dfd-bce2-12385c80ebc8, text=Romans. You can do something similar on a map of New York City: if you zoom in on the Upper East Side, there's a tiny corner that's not rich, or at least wasn't in 1993. It's called Yorkville, and that was my new home. Now I was a New York artist — in the strictly technical sense of making paintings and living in New York.\n",
      "\n",
      "I was nervous about money, because I could sense that Interleaf was on the ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=486, parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=7802050b-0b5b-4a74-9c6b-bfd8cf28ad6e, text=photorealists, and I'd taken her painting class at Harvard. I've never known a teacher more beloved by her students. Large numbers of former students kept in touch with her, including me. After I moved to New York I became her de facto studio assistant.\n",
      "\n",
      "She liked to paint on big, square canvases, 4 to 5 feet on a side. One day in late 1994 as I was stretching one of these monsters there was somet ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=483, parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=20e5d202-ef18-4992-9586-178f297968a7, text=I was right about that part. What I got wrong was the idea. I decided we should start a company to put art galleries online. I can't honestly say, after reading so many Y Combinator applications, that this was the worst startup idea ever, but it was up there. Art galleries didn't want to be online, and still don't, not the fancy ones. That's not how they sell. I wrote some software to generate web ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=472, parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=b81c7796-a4ab-435d-9eb1-6a9d6a7da7f6, text=online stores. At first this was going to be normal desktop software, which in those days meant Windows software. That was an alarming prospect, because neither of us knew how to write Windows software or wanted to learn. We lived in the Unix world. But we decided we'd at least try writing a prototype store builder on Unix. Robert wrote a shopping cart, and I wrote a new site generator for stores  ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=469, parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=6631aeb1-ee21-4de9-a90a-5b428039671d, text=is common now, but at the time it wasn't clear that it was even possible. To find out, we decided to try making a version of our store builder that you could control through the browser. A couple days later, on August 12, we had one that worked. The UI was horrible, but it proved you could build a whole store through the browser, without any client software or typing anything into the command line ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=474, parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=ba832326-ff05-4cc2-b98a-64a4b2dec7df, text=later this deal became the model for Y Combinator's. We knew founders needed something like this, because we'd needed it ourselves.\n",
      "\n",
      "At this stage I had a negative net worth, because the thousand dollars or so I had in the bank was more than counterbalanced by what I owed the government in taxes. (Had I diligently set aside the proper proportion of the money I'd made consulting for Interleaf? No,  ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=488, parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=14fe73de-2687-465e-8b2f-e6a2f5015d88, text=can make a guy running a store out of his bedroom look more legit than a big company.\n",
      "\n",
      "(If you're curious why my site looks so old-fashioned, it's because it's still made with this software. It may look clunky today, but in 1996 it was the last word in slick.)\n",
      "\n",
      "In September, Robert rebelled. \"We've been working on this for a month,\" he said, \"and it's still not done.\" This is funny in retrospect,  ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=496, parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=1a88ee7c-c50c-4ae5-9813-d514677c6e6f, text=and if you could see inside Trevor's it would look like the worst excesses of Austrian Rococo.\n",
      "\n",
      "We opened for business, with 6 stores, in January 1996. It was just as well we waited a few months, because although we worried we were late, we were actually almost fatally early. There was a lot of talk in the press then about ecommerce, but not many people actually wanted online stores. [8]\n",
      "\n",
      "There we ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=481, parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=a793e998-80c1-45c2-85ed-aad3a0f0da7a, text=a lot of startups making ecommerce software in the second half of the 90s. We were determined to be the Microsoft Word, not the Interleaf. Which meant being easy to use and inexpensive. It was lucky for us that we were poor, because that caused us to make Viaweb even more inexpensive than we realized. We charged $100 a month for a small store and $300 a month for a big one. This low price was a bi ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=476, parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=ad455333-ddaa-43be-8209-2b0c12670921, text=stores. But anything to get users.\n",
      "\n",
      "We learned a lot more about retail than we wanted to know. For example, that if you could only have a small image of a man's shirt (and all images were small then by present standards), it was better to have a closeup of the collar than a picture of the whole shirt. The reason I remember learning this was that it meant I had to rescan about 30 images of men's sh ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=488, parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=53f9700d-aeda-422c-a1a2-811d575342c6, text=at the time is that growth rate is the ultimate test of a startup. Our growth rate was fine. We had about 70 stores at the end of 1996 and about 500 at the end of 1997. I mistakenly thought the thing that mattered was the absolute number of users. And that is the thing that matters in the sense that that's how much money you're making, and if you're not making enough, you might go out of business. ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=488, parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=5cad7fb4-c7e5-48e0-a2f6-53bb58708213, text=mercy of investors for the entire life of the company. And since both we and our investors were noobs at startups, the result was a mess even by startup standards.\n",
      "\n",
      "It was a huge relief when Yahoo bought us. In principle our Viaweb stock was valuable. It was a share in a business that was profitable and growing rapidly. But it didn't feel very valuable to me; I had no idea how to value a business, ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=479, parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=594aa953-adfb-45f2-b3ba-32e1c0f3aceb, text=tried to continue my usual m.o. of programming till 3 in the morning, but fatigue combined with Yahoo's prematurely aged culture and grim cube farm in Santa Clara gradually dragged me down. After a few months it felt disconcertingly like working at Interleaf.\n",
      "\n",
      "Yahoo had given us a lot of options when they bought us. At the time I thought Yahoo was so overvalued that they'd never be worth anything, ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=477, parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=4527e8a9-b812-416d-a0ff-4fa8ec427727, text=in me. Now I realize it was because he thought I was lying. My options at that point were worth about $2 million a month. If I was leaving that kind of money on the table, it could only be to go and start some new startup, and if I did, I might take people with me. This was the height of the Internet Bubble, and Yahoo was ground zero of it. My boss was at that moment a billionaire. Leaving then to ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=467, parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=2ab7be21-72ac-42b1-9ee1-c89f8482967b, text=this problem by buying a house up in the Santa Cruz Mountains, with a beautiful view but miles from anywhere. I stuck it out for a few more months, then in desperation I went back to New York, where unless you understand about rent control you'll be surprised to hear I still had my apartment, sealed up like a tomb of my old life. Idelle was in New York at least, and there were other people trying  ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=478, parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=f2a9826c-1218-40f7-bf49-041a0d8acf63, text=photograph it and print it, blown up, on canvas, and then use that as the underpainting for a second still life, painted from the same objects (which hopefully hadn't rotted yet).\n",
      "\n",
      "Meanwhile I looked for an apartment to buy. Now I could actually choose what neighborhood to live in. Where, I asked myself and various real estate agents, is the Cambridge of New York? Aided by occasional visits to act ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=492, parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=06a7ded2-737c-4182-bc71-1346be816b75, text=this idea would have to be embodied as one, so I decided to move to Cambridge and start it. I hoped to lure Robert into working on it with me, but there I ran into a hitch. Robert was now a postdoc at MIT, and though he'd made a lot of money the last time I'd lured him into working on one of my schemes, it had also been a huge time sink. So while he agreed that it sounded like a plausible idea, he ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=477, parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=4d7b177b-12bb-46ea-b95f-5e079bb025c8, text=was replaced by \"software as a service,\" but it was current for long enough that I named this new company after it: it was going to be called Aspra.\n",
      "\n",
      "I started working on the application builder, Dan worked on network infrastructure, and the two undergrads worked on the first two services (images and phone calls). But about halfway through the summer I realized I really didn't want to run a compan ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=488, parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=18cef245-9ecb-474d-8cf6-305b13d830c7, text=new Lisp, whose parentheses I now wouldn't even have to hide. A lot of Lisp hackers dream of building a new Lisp, partly because one of the distinctive features of the language is that it has dialects, and partly, I think, because we have in our minds a Platonic form of Lisp that all existing dialects fall short of. I certainly did. So at the end of the summer Dan and I switched to working on this ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=489, parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=fafc20bf-015a-4724-bdd0-a59c5cf879f9, text=era there was a narrow channel to readers, guarded by fierce monsters known as editors. The only way to get an audience for anything you wrote was to get it published as a book, or in a newspaper or magazine. Now anyone could publish anything.\n",
      "\n",
      "This had been possible in principle since 1993, but not many people had realized it yet. I had been intimately involved with building the infrastructure of ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=480, parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=81d7cf17-0adb-4dcf-8791-71f8f7491337, text=point where I figured out what to work on, it was when I started publishing essays online. From then on I knew that whatever else I did, I'd always write essays too.\n",
      "\n",
      "I knew that online essays would be a marginal medium at first. Socially they'd seem more like rants posted by nutjobs on their GeoCities sites than the genteel and beautifully typeset compositions published in The New Yorker. But by  ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=490, parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=0f1dd2f7-8abe-4546-8887-59de7ed098e1, text=current lack of prestige, it's a sign both that there's something real to be discovered there, and that you have the right kind of motives. Impure motives are a big danger for the ambitious. If anything is going to lead you astray, it will be the desire to impress people. So while working on things that aren't prestigious doesn't guarantee you're on the right track, it at least guarantees you're n ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=472, parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=5382a1db-6f37-4b94-8c48-473519f5cb4d, text=would all invite their friends to one party. So for every guest, two thirds of the other guests would be people they didn't know but would probably like. One of the guests was someone I didn't know but would turn out to like a lot: a woman called Jessica Livingston. A couple days later I asked her out.\n",
      "\n",
      "Jessica was in charge of marketing at a Boston investment bank. This bank thought it understood ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=466, parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=c5338ecf-9972-4013-84b6-93e09f14c99f, text=founders instead of MBAs, they should let the founders remain as CEO, and so on.\n",
      "\n",
      "One of my tricks for writing essays had always been to give talks. The prospect of having to stand up in front of a group of people and tell them something that won't waste their time is a great spur to the imagination. When the Harvard Computer Society, the undergrad computer club, asked me to give a talk, I decided ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=485, parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=910c0c17-559c-4eb3-9d87-de0d23850b3a, text=one angel investment.\n",
      "\n",
      "Meanwhile I had been scheming with Robert and Trevor about projects we could work on together. I missed working with them, and it seemed like there had to be something we could collaborate on.\n",
      "\n",
      "As Jessica and I were walking home from dinner on March 11, at the corner of Garden and Walker streets, these three threads converged. Screw the VCs who were taking so long to make up ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=476, parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=01b7ac17-2d50-4a29-b4e8-34e8397a6ae5, text=were organized companies with people whose job it was to make investments, but they only did big, million dollar investments. And there were angels, who did smaller investments, but these were individuals who were usually focused on other things and made investments on the side. And neither of them helped founders enough in the beginning. We knew how helpless founders were in some respects, becaus ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=468, parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=a1c2324c-003c-469d-a6bb-bb3ed1351e9f, text=firms were organized. It never occurred to us to try to raise a fund, and if it had, we wouldn't have known where to start. [14]\n",
      "\n",
      "The most distinctive thing about YC is the batch model: to fund a bunch of startups all at once, twice a year, and then to spend three months focusing intensively on trying to help them. That part we discovered by accident, not merely implicitly but explicitly due to ou ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=485, parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=9d3b209a-79e3-4054-b11d-4d4a058f60e4, text=— on tuesdays, since I was already cooking for the thursday diners on thursdays — and after dinner we'd bring in experts on startups to give talks.\n",
      "\n",
      "We knew undergrads were deciding then about summer jobs, so in a matter of days we cooked up something we called the Summer Founders Program, and I posted an announcement on my site, inviting undergrads to apply. I had never imagined that writing essa ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=489, parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=6f49b02e-926d-4324-b17e-54a3ad8d2297, text=Sam Altman, who would later become the second president of YC. I don't think it was entirely luck that the first batch was so good. You had to be pretty bold to sign up for a weird thing like the Summer Founders Program instead of a summer job at a legit place like Microsoft or Goldman Sachs.\n",
      "\n",
      "The deal for startups was based on a combination of the deal we did with Julian ($10k for 10%) and what R ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=483, parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=cc3eb822-baba-4a71-983d-d144db43afd6, text=you not only had colleagues, but colleagues who understood the problems you were facing and could tell you how they were solving them.\n",
      "\n",
      "As YC grew, we started to notice other advantages of scale. The alumni became a tight community, dedicated to helping one another, and especially the current batch, whose shoes they remembered being in. We also noticed that the startups were becoming one another's ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=480, parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=2aba7ce3-87b7-4e49-8745-58526519a13b, text=News in it. It was originally meant to be a news aggregator for startup founders and was called Startup News, but after a few months I got tired of reading about nothing but startups. Plus it wasn't startup founders we wanted to reach. It was future startup founders. So I changed the name to Hacker News and the topic to whatever engaged one's intellectual curiosity.\n",
      "\n",
      "HN was no doubt good for YC, b ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=480, parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=e9db9a00-ab2c-43af-92fa-cebc4e272967, text=YC's internal software in Arc. But while I continued to work a good deal in Arc, I gradually stopped working on Arc, partly because I didn't have time to, and partly because it was a lot less attractive to mess around with the language now that we had all this infrastructure depending on it. So now my three projects were reduced to two: writing essays and working on YC.\n",
      "\n",
      "YC was different from othe ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=494, parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=611600f3-93b1-4f4c-a4a1-d4d08f428662, text=something Kevin Hale once said about companies: \"No one works harder than the boss.\" He meant it both descriptively and prescriptively, and it was the second part that scared me. I wanted YC to be good, so if how hard I worked set the upper bound on how hard everyone else worked, I'd better work very hard.\n",
      "\n",
      "One day in 2010, when he was visiting California for interviews, Robert Morris did somethin ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=488, parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=9b70f38f-acd1-4c79-bc49-5bb602f4b178, text=this set me thinking. It was true that on my current trajectory, YC would be the last thing I did, because it was only taking up more of my attention. It had already eaten Arc, and was in the process of eating essays too. Either YC was my life's work or I'd have to leave eventually. And it wasn't, so I would.\n",
      "\n",
      "In the summer of 2012 my mother had a stroke, and the cause turned out to be a blood clo ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=486, parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=b59b0880-c3b4-4293-bf96-64b4f4457efb, text=guard. Up till that point YC had been controlled by the original LLC we four had started. But we wanted YC to last for a long time, and to do that it couldn't be controlled by the founders. So if Sam said yes, we'd let him reorganize YC. Robert and I would retire, and Jessica and Trevor would become ordinary partners.\n",
      "\n",
      "When we asked Sam if he wanted to be president of YC, initially he said no. He  ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=497, parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=0e2c8e75-8b4b-4474-99dc-2b369707fe02, text=only takes a few hours a week.)\n",
      "\n",
      "What should I do next? Rtm's advice hadn't included anything about that. I wanted to do something completely different, so I decided I'd paint. I wanted to see how good I could get if I really focused on it. So the day after I stopped working on YC, I started painting. I was rusty and it took a while to get back into shape, but it was at least completely engaging.  ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=484, parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=61413a55-8b03-4c7a-a9c1-e4583fa86617, text=one (or at least a good one) for you, then it's getting in the way of another project that is. And at 50 there was some opportunity cost to screwing around.\n",
      "\n",
      "I started writing essays again, and wrote a bunch of new ones over the next few months. I even wrote a couple that weren't about startups. Then in March 2015 I started working on Lisp again.\n",
      "\n",
      "The distinctive thing about Lisp is that its core  ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=494, parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=67ba3912-6a00-4284-b174-e4df8013ac10, text=of computation gave it a power and elegance that other languages couldn't match. It was this that attracted me in college, though I didn't understand why at the time.\n",
      "\n",
      "McCarthy's 1960 Lisp did nothing more than interpret Lisp expressions. It was missing a lot of things you'd want in a programming language. So these had to be added, and when they were, they weren't defined using McCarthy's original ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=500, parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=50f82e98-c065-48cd-9932-402b1a772a9d, text=I decided to take a shot at it. It took 4 years, from March 26, 2015 to October 12, 2019. It was fortunate that I had a precisely defined goal, or it would have been hard to keep at it for so long.\n",
      "\n",
      "I wrote this new Lisp, called Bel, in itself in Arc. That may sound like a contradiction, but it's an indication of the sort of trickery I had to engage in to make this work. By means of an egregious c ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=471, parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=c1023af0-c084-415d-936f-20f68d2bd6de, text=more essays till Bel was done. But I told few people about Bel while I was working on it. So for years it must have seemed that I was doing nothing, when in fact I was working harder than I'd ever worked on anything. Occasionally after wrestling for hours with some gruesome bug I'd check Twitter or HN and see someone asking \"Does Paul Graham still code?\"\n",
      "\n",
      "Working on Bel was hard but satisfying. I  ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=471, parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=dbbfee51-244d-4abf-8248-da8c0b79c033, text=by birth, that seemed the obvious choice. We only meant to stay for a year, but we liked it so much that we still live there. So most of Bel was written in England.\n",
      "\n",
      "In the fall of 2019, Bel was finally finished. Like McCarthy's original Lisp, it's a spec rather than an implementation, although like McCarthy's Lisp it's a spec expressed as code.\n",
      "\n",
      "Now that I could write essays again, I wrote a bunc ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=510, parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=bfb427e0-784e-4563-b47d-83f196c70aa4, text=OSes. I went straight from batch processing to microcomputers, which made microcomputers seem all the more exciting.\n",
      "\n",
      "[2] Italian words for abstract concepts can nearly always be predicted from their English cognates (except for occasional traps like polluzione). It's the everyday words that differ. So if you string together a lot of abstract concepts with a few simple verbs, you can make a little ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=531, parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=90f71a37-d32b-402e-8927-f45ca5e4c3aa, text=In the 1990s the exponential growth in the power of commodity (i.e. Intel) processors rolled up high-end, special-purpose hardware and software companies like a bulldozer.\n",
      "\n",
      "[6] The signature style seekers at RISD weren't specifically mercenary. In the art world, money and coolness are tightly coupled. Anything expensive comes to be seen as cool, and anything seen as cool will soon become equally e ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=509, parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=9d5a1bac-0860-4840-b647-4faa760e23cc, text=were generated, not when shoppers visited them.\n",
      "\n",
      "[10] This was the first instance of what is now a familiar experience, and so was what happened next, when I read the comments and found they were full of angry people. How could I claim that Lisp was better than other languages? Weren't they all Turing complete? People who see the responses to essays I write sometimes tell me how sorry they feel fo ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=499, parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=3041116f-c976-42bd-80db-491c3b1bbaf2, text=once been much more expensive to start, and proportionally rare. Now they could be cheap and common, but the VCs' customs still reflected the old world, just as customs about writing essays still reflected the constraints of the print era.\n",
      "\n",
      "Which in turn implies that people who are independent-minded (i.e. less influenced by custom) will have an advantage in fields affected by rapid change (where  ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=502, parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=ec0b5e33-57ce-4718-8d62-8fc22edb7afc, text=the Viaweb logo had been a white V on a red circle, so I made the YC logo a white Y on an orange square.\n",
      "\n",
      "[14] YC did become a fund for a couple years starting in 2009, because it was getting so big I could no longer afford to fund it personally. But after Heroku got bought we had enough money to go back to being self-funded.\n",
      "\n",
      "[15] I've never liked the term \"deal flow,\" because it implies that the ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=498, parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=c2748c28-a904-4173-ae0e-5985506cdef0, text=misinterpretations of them on forums. Individually these two phenomena are tedious but bearable, but the combination is disastrous. You actually have to respond to the misinterpretations, because the assumption that you're present in the conversation means that not responding to any sufficiently upvoted misinterpretation reads as a tacit admission that it's correct. But that in turn encourages mor ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=316, parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5)]\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "# test using only the document splitter\n",
    "text_split = components[\"document_splitter\"](documents)\n",
    "print(text_split)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Splitting documents: 100%|██████████| 1/1 [00:00<00:00, 19.23it/s]\n",
      "Batch embedding documents: 100%|██████████| 1/1 [00:02<00:00,  2.89s/it]\n",
      "Adding embeddings to documents from batch: 1it [00:00, 2794.34it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[Document(id=e45f82df-1dd9-4d0b-87eb-70ebbc07f229, text=\n",
      "\n",
      "What I Worked On\n",
      "\n",
      "February 2021\n",
      "\n",
      "Before college the two main things I worked on, outside of school, were writing and programming. I didn't write essays. I wrote what beginning writers were supposed to write then, and probably still are: short stories. My stories were awful. They had hardly any plot, just characters with strong feelings, which I imagined made them deep.\n",
      "\n",
      "The first programs I trie ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=499, vector=[-0.07907507, 0.038137976, -0.00067343825, -0.019498399, 0.12735985, -0.03413015, 0.012397228, 0.11991674, -0.072140895, 0.09001707]..., parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=ba2beda6-43c4-4186-a678-4d9089899542, text=spectacularly loud printer.\n",
      "\n",
      "I was puzzled by the 1401. I couldn't figure out what to do with it. And in retrospect there's not much I could have done with it. The only form of input to programs was data stored on punched cards, and I didn't have any data stored on punched cards. The only other option was to do things that didn't rely on any input, like calculate approximations of pi, but I didn't ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=493, vector=[-0.02747155, -0.009451727, -0.013175614, -0.0055384915, 0.12446616, -0.014666746, 0.011794936, 0.12446616, -0.08893145, 0.04629611]..., parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=299ce3d8-df51-4005-98ac-79cca9cf2382, text=was sold as a kit by Heathkit. I remember vividly how impressed and envious I felt watching him sitting in front of it, typing programs right into the computer.\n",
      "\n",
      "Computers were expensive in those days and it took me years of nagging before I convinced my father to buy one, a TRS-80, in about 1980. The gold standard then was the Apple II, but a TRS-80 was good enough. This was when I really started ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=487, vector=[-0.0427506, 0.027133223, -0.06859026, 0.027701128, 0.146898, -0.04991251, 0.045653224, 0.14159757, -0.08165207, 0.07218699]..., parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=0286da32-4024-4161-b7a6-64a819d7791a, text=was that the other fields took up so much of the space of ideas that there wasn't much left for these supposed ultimate truths. All that seemed left for philosophy were edge cases that people in other fields felt could safely be ignored.\n",
      "\n",
      "I couldn't have put this into words when I was 18. All I knew at the time was that I kept taking philosophy courses and they kept being boring. So I decided to s ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=480, vector=[-0.053840023, 0.045960996, -0.038320728, 0.042767603, 0.15590924, -0.05443692, 0.043185428, 0.11651411, -0.13680857, 0.08995224]..., parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=cd3e25af-d049-45ef-865c-92d9b1c5cce8, text=any classes in AI at Cornell then, not even graduate classes, so I started trying to teach myself. Which meant learning Lisp, since in those days Lisp was regarded as the language of AI. The commonly used programming languages then were pretty primitive, and programmers' ideas correspondingly so. The default language at Cornell was a Pascal-like language called PL/I, and the situation was similar  ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=482, vector=[-0.04075746, 0.04185597, -0.04850335, 0.020758983, 0.11407582, -0.055009894, -0.024800928, 0.07875457, -0.12415956, 0.072050855]..., parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=c482c3e9-6e9c-4747-9c32-3a5601d00423, text=didn't make you choose a major. You could take whatever classes you liked, and choose whatever you liked to put on your degree. I of course chose \"Artificial Intelligence.\" When I got the actual physical diploma, I was dismayed to find that the quotes had been included, which made them read as scare-quotes. At the time this bothered me, but now it seems amusingly accurate, for reasons I was about  ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=479, vector=[-0.02932792, 0.039015952, -0.05156203, 0.004052559, 0.116608165, -0.09397832, -0.060385324, 0.043588825, -0.15700184, 0.06876892]..., parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=76b046f1-3378-40a9-833a-1e724f048197, text=really showed was that there's a subset of natural language that's a formal language. But a very proper subset. It was clear that there was an unbridgeable gap between what they could do and actually understanding natural language. It was not, in fact, simply a matter of teaching SHRDLU more words. That whole way of doing AI, with explicit data structures representing concepts, was not going to wo ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=479, vector=[-0.014877682, 0.06511544, -0.0448692, -0.003099517, 0.14383334, -0.08526722, -0.013216734, 0.06524139, -0.10881961, 0.061651852]..., parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=16019ccb-f37f-4ec4-9060-f2fb73a271d0, text=you learn it. The book, On Lisp, wasn't published till 1993, but I wrote much of it in grad school.\n",
      "\n",
      "Computer Science is an uneasy alliance between two halves, theory and systems. The theory people prove things, and the systems people build things. I wanted to build things. I had plenty of respect for theory — indeed, a sneaking suspicion that it was the more admirable of the two halves — but buil ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=488, vector=[-0.013353117, 0.026692593, -0.06426954, 0.011730011, 0.062087215, -0.049811624, -0.0336624, 0.043182805, -0.15243557, 0.02738821]..., parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=d68b64ba-93d8-45ab-befc-28ba8e0f09fe, text=That was what happened to systems work.\n",
      "\n",
      "I wanted not just to build things, but to build things that would last.\n",
      "\n",
      "In this dissatisfied state I went in 1988 to visit Rich Draves at CMU, where he was in grad school. One day I went to visit the Carnegie Institute, where I'd spent a lot of time as a kid. While looking at a painting there I realized something that might seem obvious, but was a big surp ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=480, vector=[0.0035078581, 0.053412426, -0.059784114, 0.07627734, 0.05091253, -0.049906474, -0.0028066675, 0.07359452, -0.14865239, 0.018535819]..., parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=d4f56b78-c01c-4d7e-91d3-094a562103bf, text=didn't just appear spontaneously — but it was as if the people who made it were a different species. They either lived long ago or were mysterious geniuses doing strange things in profiles in Life magazine. The idea of actually being able to make art, to put that verb before that noun, seemed almost miraculous.\n",
      "\n",
      "That fall I started taking art classes at Harvard. Grad students could take classes in ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=487, vector=[0.03270501, 0.077641636, -0.01036047, 0.027783241, 0.05751772, -0.034452386, -0.03057904, 0.0774669, -0.12196668, 0.024710776]..., parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=6b60e361-c826-4568-99cc-1663b562eac5, text=to get out of grad school.\n",
      "\n",
      "Then one day in April 1990 a crack appeared in the wall. I ran into professor Cheatham and he asked if I was far enough along to graduate that June. I didn't have a word of my dissertation written, but in what must have been the quickest bit of thinking in my life, I decided to take a shot at writing one in the 5 weeks or so that remained before the deadline, reusing pa ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=497, vector=[0.0015751657, 0.05071146, 0.027805021, -0.007688826, 0.0350694, -0.025954137, -0.032063447, 0.021904457, -0.10932745, -0.00969975]..., parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=2a0897c6-0e8a-4165-b23f-986fee657c75, text=the Accademia, so off to Providence I went.\n",
      "\n",
      "I'd applied for the BFA program at RISD, which meant in effect that I had to go to college again. This was not as strange as it sounds, because I was only 25, and art schools are full of people of different ages. RISD counted me as a transfer sophomore and said I had to do the foundation that summer. The foundation means the classes that everyone has to ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=488, vector=[-0.009668583, -0.00046487196, 0.012373978, -0.04883275, 0.07752953, 0.011002441, -0.044582494, -0.0047212527, -0.084944874, -0.028199406]..., parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=dd91354b-587a-40c6-ac67-f54c19176b64, text=so many stranieri attracted by the idea of studying art in Florence that the Italian students would otherwise have been outnumbered. I was in decent shape at painting and drawing from the RISD foundation that summer, but I still don't know how I managed to pass the written exam. I remember that I answered the essay question by writing about Cezanne, and that I cranked up the intellectual level as  ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=480, vector=[0.047631573, -0.029682525, -0.007579192, -0.056257267, 0.038498487, 0.023768218, -0.02930198, -0.03665919, -0.0893647, -0.029508108]..., parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=399c03e1-b888-4cf0-8048-3af3ee069728, text=that you see in 19th century studio paintings, and a nude model sitting as close to it as possible without getting burned. Except hardly anyone else painted her besides me. The rest of the students spent their time chatting or occasionally trying to imitate things they'd seen in American art magazines.\n",
      "\n",
      "Our model turned out to live just down the street from me. She made a living from a combination ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=484, vector=[0.037977736, -0.023429813, -0.054210156, -0.04045854, 0.03319989, 0.014402444, 0.012993592, -0.028085148, -0.06652229, 0.0016825822]..., parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=b369e453-34d7-49cf-b64e-05a32e0cb6d2, text=generic person, which you then modify to match the specific person you're painting. Whereas a still life you can, if you want, copy pixel by pixel from what you're seeing. You don't want to stop there, of course, or you get merely photographic accuracy, and what makes a still life interesting is that it's been through a head. You want to emphasize the visual cues that tell you, for example, that t ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=486, vector=[0.017387627, -0.011447118, -0.06267119, 0.015462681, 0.02985244, -0.044399984, 0.004035286, 0.023320246, -0.041054994, 0.010389974]..., parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=be4cfbd7-fdfb-4f9b-8111-5b252165b286, text=to notice every leaf on every bush. But when you have to paint something, you have to look more closely, and when you do there's a lot to see. You can still be noticing new things after days of trying to paint something people usually take for granted, just as you can after days of trying to write an essay about something people usually take for granted.\n",
      "\n",
      "This is not the only way to paint. I'm not ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=479, vector=[-0.044350445, 0.062140558, -0.05280855, -0.0019838319, 0.07278342, -0.055867203, -0.020958003, 0.060267914, -0.073907, 0.05786469]..., parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=a5f43a04-6644-48bf-b0d5-6667afae561b, text=software for creating documents. You mean like Microsoft Word? Exactly. That was how I learned that low end software tends to eat high end software. But Interleaf still had a few years to live yet. [5]\n",
      "\n",
      "Interleaf had done something pretty bold. Inspired by Emacs, they'd added a scripting language, and even made the scripting language a dialect of Lisp. Now they wanted a Lisp hacker to write things ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=469, vector=[-0.058567457, 0.081068136, 0.024077367, 0.0144037185, 0.09939715, -0.0729219, -0.015725838, 0.07686363, -0.049698576, 0.04651235]..., parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=40b99dcb-9841-4fa8-aeeb-68b28c3eb192, text=this time gotten a contract to publish.\n",
      "\n",
      "The good part was that I got paid huge amounts of money, especially by art student standards. In Florence, after paying my part of the rent, my budget for everything else had been $7 a day. Now I was getting paid more than 4 times that every hour, even when I was just sitting in a meeting. By living cheaply I not only managed to save enough to go back to RI ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=479, vector=[-0.066288754, 0.020486224, 0.020769367, 0.02410046, 0.06145867, -0.04666862, -0.0289472, 0.037974462, -0.035975806, 0.034943167]..., parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=85e4906d-d5f3-4e73-95fc-726e0883215b, text=it.\n",
      "\n",
      "But the most important thing I learned, and which I used in both Viaweb and Y Combinator, is that the low end eats the high end: that it's good to be the \"entry level\" option, even though that will be less prestigious, because if you're not, someone else will be, and will squash you against the ceiling. Which in turn means that prestige is a danger sign.\n",
      "\n",
      "When I left to go back to RISD the ne ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=489, vector=[-0.004602037, 0.02762098, -0.049181808, 0.026342394, 0.0909373, -0.012181605, -0.013792975, 0.0042626862, -0.06613622, 0.0154481325]..., parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=78e7028e-3467-406d-8717-a2f479977f61, text=it was more like the Accademia than not. Better organized, certainly, and a lot more expensive, but it was now becoming clear that art school did not bear the same relationship to art that medical school bore to medicine. At least not the painting department. The textile department, which my next door neighbor belonged to, seemed to be pretty rigorous. No doubt illustration and architecture were t ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=484, vector=[0.011460669, 0.049471214, -0.050929107, 0.009533009, 0.0056695887, -0.012837569, 0.002219239, 0.04522712, -0.10153424, 0.004576168]..., parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=2a87e6dd-e3c5-4290-ad5a-4f8753409c87, text=kids who \"could draw\" in high school, and now had come to what was supposed to be the best art school in the country, to learn to draw even better. They tended to be confused and demoralized by what they found at RISD, but they kept going, because painting was what they did. I was not one of the kids who could draw in high school, but at RISD I was definitely closer to their tribe than the tribe o ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=485, vector=[-0.021421997, 0.034678727, -0.044136565, 0.024826817, 0.024022901, -0.07105987, -0.0067662927, 0.097478755, -0.10882816, 0.002019642]..., parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=77dafffb-e77b-4a62-919c-d5d13791adc7, text=Romans. You can do something similar on a map of New York City: if you zoom in on the Upper East Side, there's a tiny corner that's not rich, or at least wasn't in 1993. It's called Yorkville, and that was my new home. Now I was a New York artist — in the strictly technical sense of making paintings and living in New York.\n",
      "\n",
      "I was nervous about money, because I could sense that Interleaf was on the ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=486, vector=[-0.0731452, 0.008238716, -0.0652409, 0.05916067, 0.07764457, -0.07752297, -0.011856454, 0.10853216, -0.08457604, -0.008649131]..., parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=e6d7635a-959b-4bf4-8deb-17bfc566c76f, text=photorealists, and I'd taken her painting class at Harvard. I've never known a teacher more beloved by her students. Large numbers of former students kept in touch with her, including me. After I moved to New York I became her de facto studio assistant.\n",
      "\n",
      "She liked to paint on big, square canvases, 4 to 5 feet on a side. One day in late 1994 as I was stretching one of these monsters there was somet ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=483, vector=[-0.022256354, 0.016246697, -0.09721504, 0.05488231, 0.051435597, -0.10994138, -0.021181095, 0.08142497, -0.100219876, -0.0007875707]..., parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=2617ebb0-38ae-48c7-987e-6925bc0fa5d0, text=I was right about that part. What I got wrong was the idea. I decided we should start a company to put art galleries online. I can't honestly say, after reading so many Y Combinator applications, that this was the worst startup idea ever, but it was up there. Art galleries didn't want to be online, and still don't, not the fancy ones. That's not how they sell. I wrote some software to generate web ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=472, vector=[-0.06592135, 0.040219873, -0.09019341, 0.015120991, 0.06592135, -0.10857964, 0.0021388705, 0.042938568, -0.100731865, 0.02292673]..., parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=6a7ddbab-84f0-4c58-9128-41724f93abb0, text=online stores. At first this was going to be normal desktop software, which in those days meant Windows software. That was an alarming prospect, because neither of us knew how to write Windows software or wanted to learn. We lived in the Unix world. But we decided we'd at least try writing a prototype store builder on Unix. Robert wrote a shopping cart, and I wrote a new site generator for stores  ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=469, vector=[-0.044771, 0.077939205, -0.043221936, 0.016963774, 0.0947663, -0.09500929, -0.009089364, 0.051574733, -0.09397658, 0.06463547]..., parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=0d1ccc0e-bea0-445d-a767-0307d7b6b086, text=is common now, but at the time it wasn't clear that it was even possible. To find out, we decided to try making a version of our store builder that you could control through the browser. A couple days later, on August 12, we had one that worked. The UI was horrible, but it proved you could build a whole store through the browser, without any client software or typing anything into the command line ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=474, vector=[-0.014542281, 0.08878622, -0.06298567, 0.04715505, 0.13122576, -0.072753504, -0.002707205, 0.025328998, -0.102865376, 0.06362563]..., parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=617b7fd3-5cf5-4871-b00e-55064dff92a4, text=later this deal became the model for Y Combinator's. We knew founders needed something like this, because we'd needed it ourselves.\n",
      "\n",
      "At this stage I had a negative net worth, because the thousand dollars or so I had in the bank was more than counterbalanced by what I owed the government in taxes. (Had I diligently set aside the proper proportion of the money I'd made consulting for Interleaf? No,  ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=488, vector=[-0.033374872, 0.10630641, -0.015908288, 0.04126908, 0.08253816, -0.04260476, 0.007697283, 0.114936955, -0.06216048, 0.012355038]..., parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=2106023b-5d2f-4c00-9cc7-d3fe50e9e7dd, text=can make a guy running a store out of his bedroom look more legit than a big company.\n",
      "\n",
      "(If you're curious why my site looks so old-fashioned, it's because it's still made with this software. It may look clunky today, but in 1996 it was the last word in slick.)\n",
      "\n",
      "In September, Robert rebelled. \"We've been working on this for a month,\" he said, \"and it's still not done.\" This is funny in retrospect,  ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=496, vector=[-0.027819531, 0.09070735, 0.0051304507, -0.027231796, 0.10357225, -0.060667478, -0.007942607, 0.095996976, -0.12610215, 0.027444033]..., parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=8807f9f8-8f73-42ba-96d2-051f42bb37d4, text=and if you could see inside Trevor's it would look like the worst excesses of Austrian Rococo.\n",
      "\n",
      "We opened for business, with 6 stores, in January 1996. It was just as well we waited a few months, because although we worried we were late, we were actually almost fatally early. There was a lot of talk in the press then about ecommerce, but not many people actually wanted online stores. [8]\n",
      "\n",
      "There we ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=481, vector=[-0.04253237, 0.07392454, 0.013313501, -0.024453698, 0.13677329, -0.055121437, 0.014834814, 0.08094351, -0.10786029, 0.035931963]..., parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=f61c4edd-89c1-4eaa-ac2e-d9ea2a9b5fbb, text=a lot of startups making ecommerce software in the second half of the 90s. We were determined to be the Microsoft Word, not the Interleaf. Which meant being easy to use and inexpensive. It was lucky for us that we were poor, because that caused us to make Viaweb even more inexpensive than we realized. We charged $100 a month for a small store and $300 a month for a big one. This low price was a bi ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=476, vector=[-0.010721936, 0.06648869, -0.06033468, 0.025472527, 0.10607737, -0.10753658, -0.008834494, 0.10461818, -0.098210394, 0.036226187]..., parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=e4b2804e-9690-458d-98ab-32745c3446f9, text=stores. But anything to get users.\n",
      "\n",
      "We learned a lot more about retail than we wanted to know. For example, that if you could only have a small image of a man's shirt (and all images were small then by present standards), it was better to have a closeup of the collar than a picture of the whole shirt. The reason I remember learning this was that it meant I had to rescan about 30 images of men's sh ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=488, vector=[0.03937673, 0.07387383, -0.05095812, 0.048363887, 0.13774134, -0.07504741, -0.005539765, 0.12983511, -0.08394192, 0.030806499]..., parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=cf9a1670-f508-4cd4-b9f6-27afc5b513e0, text=at the time is that growth rate is the ultimate test of a startup. Our growth rate was fine. We had about 70 stores at the end of 1996 and about 500 at the end of 1997. I mistakenly thought the thing that mattered was the absolute number of users. And that is the thing that matters in the sense that that's how much money you're making, and if you're not making enough, you might go out of business. ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=488, vector=[0.0011247798, -0.00021909054, -0.058989327, 0.075708225, 0.09722198, -0.07545587, 0.055077735, 0.10277392, 0.00029425658, 0.07236445]..., parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=0fc2396d-751e-4cbe-8d84-48e9eafb4d46, text=mercy of investors for the entire life of the company. And since both we and our investors were noobs at startups, the result was a mess even by startup standards.\n",
      "\n",
      "It was a huge relief when Yahoo bought us. In principle our Viaweb stock was valuable. It was a share in a business that was profitable and growing rapidly. But it didn't feel very valuable to me; I had no idea how to value a business, ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=479, vector=[0.02191397, 0.019155568, -0.099425055, 0.081097014, 0.091640234, -0.07662227, 0.056026205, 0.0729444, -0.08054533, 0.08189388]..., parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=89720d2c-bf01-4a65-ba63-ffa83adb0619, text=tried to continue my usual m.o. of programming till 3 in the morning, but fatigue combined with Yahoo's prematurely aged culture and grim cube farm in Santa Clara gradually dragged me down. After a few months it felt disconcertingly like working at Interleaf.\n",
      "\n",
      "Yahoo had given us a lot of options when they bought us. At the time I thought Yahoo was so overvalued that they'd never be worth anything, ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=477, vector=[0.02318651, 0.014824535, -0.04591898, 0.0915958, 0.07621884, -0.04352768, 0.029891226, 0.102250695, -0.11242128, 0.08336247]..., parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=8ac02054-3e3b-4c09-acf8-24e0f9a98a8d, text=in me. Now I realize it was because he thought I was lying. My options at that point were worth about $2 million a month. If I was leaving that kind of money on the table, it could only be to go and start some new startup, and if I did, I might take people with me. This was the height of the Internet Bubble, and Yahoo was ground zero of it. My boss was at that moment a billionaire. Leaving then to ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=467, vector=[0.0074208225, 0.009954286, -0.061677538, 0.10111993, 0.033071652, -0.041971955, 0.02246936, 0.12591586, -0.0906894, 0.07944691]..., parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=97c49b68-1f3e-4357-b7a2-e2878eaf62ce, text=this problem by buying a house up in the Santa Cruz Mountains, with a beautiful view but miles from anywhere. I stuck it out for a few more months, then in desperation I went back to New York, where unless you understand about rent control you'll be surprised to hear I still had my apartment, sealed up like a tomb of my old life. Idelle was in New York at least, and there were other people trying  ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=478, vector=[-0.039481334, -0.013633448, 0.004927327, 0.06274992, 0.041990075, -0.113959625, 0.0427427, 0.07745742, -0.06666983, 0.007632065]..., parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=cee605b5-a1fe-44d7-9122-9d5eda32b0c2, text=photograph it and print it, blown up, on canvas, and then use that as the underpainting for a second still life, painted from the same objects (which hopefully hadn't rotted yet).\n",
      "\n",
      "Meanwhile I looked for an apartment to buy. Now I could actually choose what neighborhood to live in. Where, I asked myself and various real estate agents, is the Cambridge of New York? Aided by occasional visits to act ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=492, vector=[-0.040362474, 0.024286918, -0.02630957, 0.013517051, 0.088030644, -0.08724573, -0.011758552, 0.04099644, -0.05781162, -0.007018904]..., parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=d09c5bcb-e3ff-4bae-ba4c-a36793b37c26, text=this idea would have to be embodied as one, so I decided to move to Cambridge and start it. I hoped to lure Robert into working on it with me, but there I ran into a hitch. Robert was now a postdoc at MIT, and though he'd made a lot of money the last time I'd lured him into working on one of my schemes, it had also been a huge time sink. So while he agreed that it sounded like a plausible idea, he ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=477, vector=[-0.038379394, 0.024685407, -0.013724181, -0.018797146, 0.052632008, -0.08491176, -0.050276704, 0.022616968, -0.04571707, -0.00016159695]..., parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=40d71c28-c6e7-4004-be79-eccafc30260b, text=was replaced by \"software as a service,\" but it was current for long enough that I named this new company after it: it was going to be called Aspra.\n",
      "\n",
      "I started working on the application builder, Dan worked on network infrastructure, and the two undergrads worked on the first two services (images and phone calls). But about halfway through the summer I realized I really didn't want to run a compan ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=488, vector=[-0.031537034, 0.019595249, -0.10691024, -0.02616174, 0.09547472, -0.04898812, -0.092318036, 0.036897436, -0.06897051, 0.031030772]..., parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=fe845a49-e1dc-4ac9-8684-9e5d682c27cf, text=new Lisp, whose parentheses I now wouldn't even have to hide. A lot of Lisp hackers dream of building a new Lisp, partly because one of the distinctive features of the language is that it has dialects, and partly, I think, because we have in our minds a Platonic form of Lisp that all existing dialects fall short of. I certainly did. So at the end of the summer Dan and I switched to working on this ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=489, vector=[0.003908867, 0.024017368, -0.08921155, -0.027490288, 0.12637326, -0.08135719, -0.0706307, 0.07080654, -0.08885986, 0.034699894]..., parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=0c5dbd48-7cf1-4773-9573-b72c2eb23f1d, text=era there was a narrow channel to readers, guarded by fierce monsters known as editors. The only way to get an audience for anything you wrote was to get it published as a book, or in a newspaper or magazine. Now anyone could publish anything.\n",
      "\n",
      "This had been possible in principle since 1993, but not many people had realized it yet. I had been intimately involved with building the infrastructure of ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=480, vector=[0.024288623, 0.024016058, -0.016974779, 0.020033572, 0.16099541, -0.08728156, -0.05875303, 0.10957137, -0.083768494, 0.0407637]..., parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=86e46286-9f19-4487-b091-0478ea54dd3b, text=point where I figured out what to work on, it was when I started publishing essays online. From then on I knew that whatever else I did, I'd always write essays too.\n",
      "\n",
      "I knew that online essays would be a marginal medium at first. Socially they'd seem more like rants posted by nutjobs on their GeoCities sites than the genteel and beautifully typeset compositions published in The New Yorker. But by  ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=490, vector=[0.042509366, -0.0092501845, -0.04260892, 0.014858368, 0.13910949, -0.07532886, -0.023511231, 0.10711962, -0.10904432, 0.0015617464]..., parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=4985d12d-835e-4195-9200-0b1eae59830f, text=current lack of prestige, it's a sign both that there's something real to be discovered there, and that you have the right kind of motives. Impure motives are a big danger for the ambitious. If anything is going to lead you astray, it will be the desire to impress people. So while working on things that aren't prestigious doesn't guarantee you're on the right track, it at least guarantees you're n ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=472, vector=[0.04529276, -0.009341214, 0.020823296, 0.04124518, 0.08416291, -0.05301996, 0.008789271, 0.13119513, -0.0527189, 0.014517768]..., parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=0bd7bf59-c0a4-4661-9964-3e171da85103, text=would all invite their friends to one party. So for every guest, two thirds of the other guests would be people they didn't know but would probably like. One of the guests was someone I didn't know but would turn out to like a lot: a woman called Jessica Livingston. A couple days later I asked her out.\n",
      "\n",
      "Jessica was in charge of marketing at a Boston investment bank. This bank thought it understood ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=466, vector=[0.01819799, -0.03182732, -0.026231475, 0.062918745, 0.08346239, -0.08806171, -0.01436522, 0.17820846, -0.001847395, 0.07033899]..., parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=e63e1a32-7b24-42be-a9a5-804dea37e5cb, text=founders instead of MBAs, they should let the founders remain as CEO, and so on.\n",
      "\n",
      "One of my tricks for writing essays had always been to give talks. The prospect of having to stand up in front of a group of people and tell them something that won't waste their time is a great spur to the imagination. When the Harvard Computer Society, the undergrad computer club, asked me to give a talk, I decided ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=485, vector=[0.038747437, 0.017834086, -0.0074455705, 0.07993263, 0.13279338, -0.053790938, 0.009646765, 0.16397095, 0.0009131549, 0.030279456]..., parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=ad992581-6611-44c8-844c-14fa8ee57cd4, text=one angel investment.\n",
      "\n",
      "Meanwhile I had been scheming with Robert and Trevor about projects we could work on together. I missed working with them, and it seemed like there had to be something we could collaborate on.\n",
      "\n",
      "As Jessica and I were walking home from dinner on March 11, at the corner of Garden and Walker streets, these three threads converged. Screw the VCs who were taking so long to make up ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=476, vector=[0.018336639, 0.019985966, -0.043496966, 0.061607227, 0.13427465, -0.011852519, 0.03803155, 0.11525887, -0.011901028, 0.030561065]..., parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=1ab309b6-95ac-4769-9994-e15d8f04057f, text=were organized companies with people whose job it was to make investments, but they only did big, million dollar investments. And there were angels, who did smaller investments, but these were individuals who were usually focused on other things and made investments on the side. And neither of them helped founders enough in the beginning. We knew how helpless founders were in some respects, becaus ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=468, vector=[0.0064024017, 0.055864017, -0.03715842, 0.09555022, 0.09731967, -0.036368486, 0.0397178, 0.08373283, -0.006516942, 0.04101329]..., parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=8932a87b-c3ab-4270-9fc1-dfc3a3c4b6bf, text=firms were organized. It never occurred to us to try to raise a fund, and if it had, we wouldn't have known where to start. [14]\n",
      "\n",
      "The most distinctive thing about YC is the batch model: to fund a bunch of startups all at once, twice a year, and then to spend three months focusing intensively on trying to help them. That part we discovered by accident, not merely implicitly but explicitly due to ou ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=485, vector=[-0.013621022, 0.03989731, -0.032603923, 0.08443025, 0.093886934, -0.068112835, -0.032356687, 0.09821352, 0.002881816, 0.013698283]..., parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=ceddd78b-00c8-441a-91a0-290eee1d116b, text=— on tuesdays, since I was already cooking for the thursday diners on thursdays — and after dinner we'd bring in experts on startups to give talks.\n",
      "\n",
      "We knew undergrads were deciding then about summer jobs, so in a matter of days we cooked up something we called the Summer Founders Program, and I posted an announcement on my site, inviting undergrads to apply. I had never imagined that writing essa ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=489, vector=[0.0194155, 0.014942945, 0.0015044232, 0.04013384, 0.1173192, -0.035303798, 0.016301394, 0.107595555, 0.009818968, 0.047792003]..., parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=887b2772-2791-42e9-9bd9-2e7215008fbc, text=Sam Altman, who would later become the second president of YC. I don't think it was entirely luck that the first batch was so good. You had to be pretty bold to sign up for a weird thing like the Summer Founders Program instead of a summer job at a legit place like Microsoft or Goldman Sachs.\n",
      "\n",
      "The deal for startups was based on a combination of the deal we did with Julian ($10k for 10%) and what R ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=483, vector=[-0.012571322, 0.010419402, 0.006970003, 0.074874125, 0.14126717, -0.01348114, 0.03958898, 0.06405124, -0.03882948, 0.0021855428]..., parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=e9e5a7a0-25a7-4822-baf2-7d281eb01535, text=you not only had colleagues, but colleagues who understood the problems you were facing and could tell you how they were solving them.\n",
      "\n",
      "As YC grew, we started to notice other advantages of scale. The alumni became a tight community, dedicated to helping one another, and especially the current batch, whose shoes they remembered being in. We also noticed that the startups were becoming one another's ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=480, vector=[-0.0061149863, 0.005913606, 0.008845236, 0.09313058, 0.15850168, -0.05743982, 0.033336163, 0.05520915, -0.056107614, 0.10416002]..., parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=f549e567-afa8-4901-a6e1-950e545f346d, text=News in it. It was originally meant to be a news aggregator for startup founders and was called Startup News, but after a few months I got tired of reading about nothing but startups. Plus it wasn't startup founders we wanted to reach. It was future startup founders. So I changed the name to Hacker News and the topic to whatever engaged one's intellectual curiosity.\n",
      "\n",
      "HN was no doubt good for YC, b ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=480, vector=[-0.0348371, 0.01800779, -0.02318162, 0.07921704, 0.13635908, -0.049208842, 0.018252112, 0.07306593, -0.027521884, 0.14808641]..., parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=d4c8e8ea-e69f-43d3-a661-c7cf0d468619, text=YC's internal software in Arc. But while I continued to work a good deal in Arc, I gradually stopped working on Arc, partly because I didn't have time to, and partly because it was a lot less attractive to mess around with the language now that we had all this infrastructure depending on it. So now my three projects were reduced to two: writing essays and working on YC.\n",
      "\n",
      "YC was different from othe ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=494, vector=[-0.015509707, 0.056496896, -0.07327663, 0.11170068, 0.12081407, -0.0060037575, 0.007458514, 0.07807963, -0.034852583, 0.10024735]..., parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=173eb629-59d5-41fa-8888-30c203cd000b, text=something Kevin Hale once said about companies: \"No one works harder than the boss.\" He meant it both descriptively and prescriptively, and it was the second part that scared me. I wanted YC to be good, so if how hard I worked set the upper bound on how hard everyone else worked, I'd better work very hard.\n",
      "\n",
      "One day in 2010, when he was visiting California for interviews, Robert Morris did somethin ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=488, vector=[0.046288613, 0.04556893, -0.0457325, 0.11979428, 0.074585184, -0.011457658, 0.008358121, 0.09061446, -0.04452212, 0.061369214]..., parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=0e6f2a93-736a-4b41-a7fe-347d59f419d4, text=this set me thinking. It was true that on my current trajectory, YC would be the last thing I did, because it was only taking up more of my attention. It had already eaten Arc, and was in the process of eating essays too. Either YC was my life's work or I'd have to leave eventually. And it wasn't, so I would.\n",
      "\n",
      "In the summer of 2012 my mother had a stroke, and the cause turned out to be a blood clo ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=486, vector=[0.055007536, 0.05108489, -0.03557536, 0.10512685, 0.091729514, 0.04366204, 0.030898362, 0.08044437, -0.0019330336, 0.044205178]..., parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=fdde9aff-afff-4a15-bf1b-bc4fcb060752, text=guard. Up till that point YC had been controlled by the original LLC we four had started. But we wanted YC to last for a long time, and to do that it couldn't be controlled by the founders. So if Sam said yes, we'd let him reorganize YC. Robert and I would retire, and Jessica and Trevor would become ordinary partners.\n",
      "\n",
      "When we asked Sam if he wanted to be president of YC, initially he said no. He  ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=497, vector=[0.031589653, 0.07312294, -0.029526517, 0.11695612, 0.0784668, 0.028139818, 0.00945322, 0.042581767, -0.026600922, 0.10058632]..., parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=0a3b69ea-9630-46ee-8758-c7ffd67508b6, text=only takes a few hours a week.)\n",
      "\n",
      "What should I do next? Rtm's advice hadn't included anything about that. I wanted to do something completely different, so I decided I'd paint. I wanted to see how good I could get if I really focused on it. So the day after I stopped working on YC, I started painting. I was rusty and it took a while to get back into shape, but it was at least completely engaging.  ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=484, vector=[-0.065562546, 0.087477095, -0.034773502, -0.004369327, 0.113195, -0.032690715, 0.015726559, 0.101301976, -0.06628699, 0.08210411]..., parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=3c355ef4-954a-4d70-96c2-fe009d58cc28, text=one (or at least a good one) for you, then it's getting in the way of another project that is. And at 50 there was some opportunity cost to screwing around.\n",
      "\n",
      "I started writing essays again, and wrote a bunch of new ones over the next few months. I even wrote a couple that weren't about startups. Then in March 2015 I started working on Lisp again.\n",
      "\n",
      "The distinctive thing about Lisp is that its core  ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=494, vector=[-0.03649605, 0.022266192, -0.03743824, -0.009304137, 0.11012551, -0.05495191, 0.015560009, 0.052291606, -0.04680473, 0.0361358]..., parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=e98f9b56-20e8-4fe3-9078-ae762bfd6dc4, text=of computation gave it a power and elegance that other languages couldn't match. It was this that attracted me in college, though I didn't understand why at the time.\n",
      "\n",
      "McCarthy's 1960 Lisp did nothing more than interpret Lisp expressions. It was missing a lot of things you'd want in a programming language. So these had to be added, and when they were, they weren't defined using McCarthy's original ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=500, vector=[-0.000718632, -0.011908207, -0.0858567, -0.022748621, 0.009989276, -0.048313737, -0.050480273, 0.0021587978, -0.09241821, 0.03314799]..., parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=af2a8d8b-4003-416a-b104-a74e5c43d7ab, text=I decided to take a shot at it. It took 4 years, from March 26, 2015 to October 12, 2019. It was fortunate that I had a precisely defined goal, or it would have been hard to keep at it for so long.\n",
      "\n",
      "I wrote this new Lisp, called Bel, in itself in Arc. That may sound like a contradiction, but it's an indication of the sort of trickery I had to engage in to make this work. By means of an egregious c ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=471, vector=[0.04828542, 0.04047009, -0.038077023, 0.0536471, -0.013017979, -0.080152586, -0.029913332, 0.036320087, -0.069429226, 0.029095449]..., parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=2102f567-7b11-4cd2-a059-a373acaba2ac, text=more essays till Bel was done. But I told few people about Bel while I was working on it. So for years it must have seemed that I was doing nothing, when in fact I was working harder than I'd ever worked on anything. Occasionally after wrestling for hours with some gruesome bug I'd check Twitter or HN and see someone asking \"Does Paul Graham still code?\"\n",
      "\n",
      "Working on Bel was hard but satisfying. I  ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=471, vector=[0.06559414, 0.0266457, -0.07093248, 0.04206247, 0.038472895, -0.08860424, -0.03604916, 0.14063776, -0.10050813, 0.008981611]..., parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=2474af54-91a4-42ad-b138-bb5c948cb3e6, text=by birth, that seemed the obvious choice. We only meant to stay for a year, but we liked it so much that we still live there. So most of Bel was written in England.\n",
      "\n",
      "In the fall of 2019, Bel was finally finished. Like McCarthy's original Lisp, it's a spec rather than an implementation, although like McCarthy's Lisp it's a spec expressed as code.\n",
      "\n",
      "Now that I could write essays again, I wrote a bunc ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=510, vector=[0.032789003, 0.0062919892, -0.037415467, -0.012508273, 0.102085, -0.038189348, -0.0841175, 0.05790649, -0.1267146, -0.02301455]..., parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=7e1547e2-414d-4ab1-a33e-5389508e2a92, text=OSes. I went straight from batch processing to microcomputers, which made microcomputers seem all the more exciting.\n",
      "\n",
      "[2] Italian words for abstract concepts can nearly always be predicted from their English cognates (except for occasional traps like polluzione). It's the everyday words that differ. So if you string together a lot of abstract concepts with a few simple verbs, you can make a little ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=531, vector=[-0.039298795, 0.052578613, -0.021541689, 0.016549084, 0.1393536, -0.06795347, -0.053828876, 0.03747409, -0.037744418, 0.006044344]..., parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=067b820e-095b-4414-a531-10e94ea541ee, text=In the 1990s the exponential growth in the power of commodity (i.e. Intel) processors rolled up high-end, special-purpose hardware and software companies like a bulldozer.\n",
      "\n",
      "[6] The signature style seekers at RISD weren't specifically mercenary. In the art world, money and coolness are tightly coupled. Anything expensive comes to be seen as cool, and anything seen as cool will soon become equally e ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=509, vector=[0.041010115, 0.0332491, -0.03515897, 0.09604909, 0.18737559, -0.072227806, -0.04118374, 0.11202255, -0.04528128, 0.083478674]..., parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=d31860e3-3090-4463-a5f9-7bedf951a334, text=were generated, not when shoppers visited them.\n",
      "\n",
      "[10] This was the first instance of what is now a familiar experience, and so was what happened next, when I read the comments and found they were full of angry people. How could I claim that Lisp was better than other languages? Weren't they all Turing complete? People who see the responses to essays I write sometimes tell me how sorry they feel fo ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=499, vector=[0.04887616, 0.0110047255, -0.055723544, 0.09309745, 0.15192424, -0.04789796, -0.00073891884, 0.07522004, -0.04459233, 0.071577094]..., parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=46c58b5b-c468-420f-a4db-a0f64d4d5f96, text=once been much more expensive to start, and proportionally rare. Now they could be cheap and common, but the VCs' customs still reflected the old world, just as customs about writing essays still reflected the constraints of the print era.\n",
      "\n",
      "Which in turn implies that people who are independent-minded (i.e. less influenced by custom) will have an advantage in fields affected by rapid change (where  ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=502, vector=[0.048474856, -0.016810106, -0.035266917, 0.11437733, 0.16000476, 0.026107123, 0.007821845, 0.07485643, -0.04216249, 0.09351908]..., parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=5c522ff7-1b96-44e2-86bd-24abd57763b1, text=the Viaweb logo had been a white V on a red circle, so I made the YC logo a white Y on an orange square.\n",
      "\n",
      "[14] YC did become a fund for a couple years starting in 2009, because it was getting so big I could no longer afford to fund it personally. But after Heroku got bought we had enough money to go back to being self-funded.\n",
      "\n",
      "[15] I've never liked the term \"deal flow,\" because it implies that the ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=498, vector=[0.0046567502, 0.03544001, -0.05263933, 0.08552632, 0.16003431, 0.0023787636, -0.0021005764, -0.005416779, -0.045786478, 0.13591495]..., parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=357e855c-f5bb-4865-995c-6de552604088, text=misinterpretations of them on forums. Individually these two phenomena are tedious but bearable, but the combination is disastrous. You actually have to respond to the misinterpretations, because the assumption that you're present in the conversation means that not responding to any sufficiently upvoted misinterpretation reads as a tacit admission that it's correct. But that in turn encourages mor ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=316, vector=[0.022537222, 0.06692978, -0.074482985, 0.06011091, 0.07993808, -0.013156911, 0.011373516, 0.014214709, -0.07151066, 0.062243994]..., parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5)]\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "# test the whole data transformer\n",
    "embeddings = data_transformer(documents)\n",
    "print(embeddings)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'Sequential__0_1_1.vectorizer_1.vectorizer.model_client_1.batch_embedder_': Sequential(\n",
       "   (0): DocumentSplitter(split_by=word, split_length=400, split_overlap=200)\n",
       "   (1): ToEmbeddings(\n",
       "     batch_size=100\n",
       "     (vectorizer): Embedder(\n",
       "       model_kwargs={'model': 'text-embedding-3-small', 'dimensions': 256, 'encoding_format': 'float'}, \n",
       "       (model_client): OpenAIClient()\n",
       "     )\n",
       "     (batch_embedder): BatchEmbedder(\n",
       "       (embedder): Embedder(\n",
       "         model_kwargs={'model': 'text-embedding-3-small', 'dimensions': 256, 'encoding_format': 'float'}, \n",
       "         (model_client): OpenAIClient()\n",
       "       )\n",
       "     )\n",
       "   )\n",
       " )}"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "db.register_transformer(data_transformer)\n",
    "db.transformer_setups"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Splitting documents: 100%|██████████| 1/1 [00:00<00:00, 20.65it/s]\n",
      "Batch embedding documents: 100%|██████████| 1/1 [00:00<00:00,  1.13it/s]\n",
      "Adding embeddings to documents from batch: 1it [00:00, 15887.52it/s]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'Sequential__0_1_1.vectorizer_1.vectorizer.model_client_1.batch_embedder_'"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "db.transform_data(transformer=data_transformer)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "68 <class 'list'> [-0.07920883, 0.038013875, -0.00059247564, -0.01969087, 0.1272431, -0.03413296, 0.012310769, 0.11960851, -0.07208321, 0.089897245]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{256}"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "keys = list(db.transformed_documents.keys())\n",
    "documents = db.transformed_documents[keys[0]]\n",
    "vectors = [doc.vector for doc in documents]\n",
    "print(len(vectors), type(vectors), vectors[0][0:10])\n",
    "\n",
    "# check if all embeddings are the same length\n",
    "dimensions = set([len(vector) for vector in vectors])\n",
    "dimensions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{531, 316, 466, 467, 468, 469, 471, 472, 474, 476, 477, 478, 479, 480, 481, 482, 483, 484, 485, 486, 487, 488, 489, 490, 492, 493, 494, 496, 497, 498, 499, 500, 502, 509, 510}\n"
     ]
    }
   ],
   "source": [
    "# check the length of all documents,text \n",
    "lengths = set([doc.estimated_num_tokens for doc in documents])\n",
    "print(lengths)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0\n"
     ]
    }
   ],
   "source": [
    "total = 0\n",
    "for doc in documents:\n",
    "    if len(doc.vector) != 256:\n",
    "        print(doc)\n",
    "        total+=1\n",
    "print(total)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# save the db states, including the original documents with len 1, and transformed documents\n",
    "db.save_state(\"developer_notes/db_states.pkl\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LocalDB(documents=[Document(id=024146ce-84e5-43a2-a1fe-e5385ad9aab5, text=\n",
       "\n",
       "What I Worked On\n",
       "\n",
       "February 2021\n",
       "\n",
       "Before college the two main things I worked on, outside of school, were writing and programming. I didn't write essays. I wrote what beginning writers were supposed to write then, and probably still are: short stories. My stories were awful. They had hardly any plot, just characters with strong feelings, which I imagined made them deep.\n",
       "\n",
       "The first programs I trie ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=16534)], transformed_documents={'Sequential__0_1_1.vectorizer_1.vectorizer.model_client_1.batch_embedder_': [Document(id=e215f93d-9f3d-41de-8052-24bff73be6b2, text=\n",
       "\n",
       "What I Worked On\n",
       "\n",
       "February 2021\n",
       "\n",
       "Before college the two main things I worked on, outside of school, were writing and programming. I didn't write essays. I wrote what beginning writers were supposed to write then, and probably still are: short stories. My stories were awful. They had hardly any plot, just characters with strong feelings, which I imagined made them deep.\n",
       "\n",
       "The first programs I trie ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=499, vector=[-0.07920883, 0.038013875, -0.00059247564, -0.01969087, 0.1272431, -0.03413296, 0.012310769, 0.11960851, -0.07208321, 0.089897245]..., parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=c7c4ac63-6883-4cc3-9139-8938dace87bb, text=spectacularly loud printer.\n",
       "\n",
       "I was puzzled by the 1401. I couldn't figure out what to do with it. And in retrospect there's not much I could have done with it. The only form of input to programs was data stored on punched cards, and I didn't have any data stored on punched cards. The only other option was to do things that didn't rely on any input, like calculate approximations of pi, but I didn't ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=493, vector=[-0.02747155, -0.009451727, -0.013175614, -0.0055384915, 0.12446616, -0.014666746, 0.011794936, 0.12446616, -0.08893145, 0.04629611]..., parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=8e9469cb-d1cb-45cd-8fbe-8d1460dab342, text=was sold as a kit by Heathkit. I remember vividly how impressed and envious I felt watching him sitting in front of it, typing programs right into the computer.\n",
       "\n",
       "Computers were expensive in those days and it took me years of nagging before I convinced my father to buy one, a TRS-80, in about 1980. The gold standard then was the Apple II, but a TRS-80 was good enough. This was when I really started ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=487, vector=[-0.0427506, 0.027133223, -0.06859026, 0.027701128, 0.146898, -0.04991251, 0.045653224, 0.14159757, -0.08165207, 0.07218699]..., parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=d6977d5d-15c8-44d5-9c8c-72544c027e9b, text=was that the other fields took up so much of the space of ideas that there wasn't much left for these supposed ultimate truths. All that seemed left for philosophy were edge cases that people in other fields felt could safely be ignored.\n",
       "\n",
       "I couldn't have put this into words when I was 18. All I knew at the time was that I kept taking philosophy courses and they kept being boring. So I decided to s ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=480, vector=[-0.053840023, 0.045960996, -0.038320728, 0.042767603, 0.15590924, -0.05443692, 0.043185428, 0.11651411, -0.13680857, 0.08995224]..., parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=83b336e2-2ec3-41e6-8272-94d2999173ff, text=any classes in AI at Cornell then, not even graduate classes, so I started trying to teach myself. Which meant learning Lisp, since in those days Lisp was regarded as the language of AI. The commonly used programming languages then were pretty primitive, and programmers' ideas correspondingly so. The default language at Cornell was a Pascal-like language called PL/I, and the situation was similar  ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=482, vector=[-0.040745128, 0.041844063, -0.048522204, 0.020724786, 0.11400746, -0.055003103, -0.024768304, 0.07878518, -0.12420783, 0.072022505]..., parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=c39ff89b-830c-4f86-ad8f-0c521632102a, text=didn't make you choose a major. You could take whatever classes you liked, and choose whatever you liked to put on your degree. I of course chose \"Artificial Intelligence.\" When I got the actual physical diploma, I was dismayed to find that the quotes had been included, which made them read as scare-quotes. At the time this bothered me, but now it seems amusingly accurate, for reasons I was about  ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=479, vector=[-0.02932792, 0.039015952, -0.05156203, 0.004052559, 0.116608165, -0.09397832, -0.060385324, 0.043588825, -0.15700184, 0.06876892]..., parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=896a2a42-a775-4a04-9df9-86a15e4d198c, text=really showed was that there's a subset of natural language that's a formal language. But a very proper subset. It was clear that there was an unbridgeable gap between what they could do and actually understanding natural language. It was not, in fact, simply a matter of teaching SHRDLU more words. That whole way of doing AI, with explicit data structures representing concepts, was not going to wo ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=479, vector=[-0.014877682, 0.06511544, -0.0448692, -0.003099517, 0.14383334, -0.08526722, -0.013216734, 0.06524139, -0.10881961, 0.061651852]..., parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=874fb5cc-a7c3-4679-81dd-7f69f917278e, text=you learn it. The book, On Lisp, wasn't published till 1993, but I wrote much of it in grad school.\n",
       "\n",
       "Computer Science is an uneasy alliance between two halves, theory and systems. The theory people prove things, and the systems people build things. I wanted to build things. I had plenty of respect for theory — indeed, a sneaking suspicion that it was the more admirable of the two halves — but buil ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=488, vector=[-0.013353117, 0.026692593, -0.06426954, 0.011730011, 0.062087215, -0.049811624, -0.0336624, 0.043182805, -0.15243557, 0.02738821]..., parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=6ac96a10-cba6-47e7-bf54-40148f554c2e, text=That was what happened to systems work.\n",
       "\n",
       "I wanted not just to build things, but to build things that would last.\n",
       "\n",
       "In this dissatisfied state I went in 1988 to visit Rich Draves at CMU, where he was in grad school. One day I went to visit the Carnegie Institute, where I'd spent a lot of time as a kid. While looking at a painting there I realized something that might seem obvious, but was a big surp ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=480, vector=[0.0035832168, 0.053521562, -0.059830766, 0.07632004, 0.050839387, -0.04989453, -0.0028326653, 0.073637865, -0.14873874, 0.018500904]..., parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=536a1b4c-7e4c-42b9-ba0e-a5c5413f4182, text=didn't just appear spontaneously — but it was as if the people who made it were a different species. They either lived long ago or were mysterious geniuses doing strange things in profiles in Life magazine. The idea of actually being able to make art, to put that verb before that noun, seemed almost miraculous.\n",
       "\n",
       "That fall I started taking art classes at Harvard. Grad students could take classes in ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=487, vector=[0.03270501, 0.077641636, -0.01036047, 0.027783241, 0.05751772, -0.034452386, -0.03057904, 0.0774669, -0.12196668, 0.024710776]..., parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=61bd45ac-2d37-4527-babd-06de79809547, text=to get out of grad school.\n",
       "\n",
       "Then one day in April 1990 a crack appeared in the wall. I ran into professor Cheatham and he asked if I was far enough along to graduate that June. I didn't have a word of my dissertation written, but in what must have been the quickest bit of thinking in my life, I decided to take a shot at writing one in the 5 weeks or so that remained before the deadline, reusing pa ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=497, vector=[0.001590877, 0.050768893, 0.027792072, -0.0077030105, 0.03501495, -0.025968958, -0.03203673, 0.021933053, -0.10938692, -0.009748797]..., parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=d5fde0d4-1b3a-42b4-a852-0335a1c2840c, text=the Accademia, so off to Providence I went.\n",
       "\n",
       "I'd applied for the BFA program at RISD, which meant in effect that I had to go to college again. This was not as strange as it sounds, because I was only 25, and art schools are full of people of different ages. RISD counted me as a transfer sophomore and said I had to do the foundation that summer. The foundation means the classes that everyone has to ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=488, vector=[-0.009668583, -0.00046487196, 0.012373978, -0.04883275, 0.07752953, 0.011002441, -0.044582494, -0.0047212527, -0.084944874, -0.028199406]..., parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=4fcdd60f-82f8-4f6c-9ec0-52c6a9e45a94, text=so many stranieri attracted by the idea of studying art in Florence that the Italian students would otherwise have been outnumbered. I was in decent shape at painting and drawing from the RISD foundation that summer, but I still don't know how I managed to pass the written exam. I remember that I answered the essay question by writing about Cezanne, and that I cranked up the intellectual level as  ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=480, vector=[0.047631573, -0.029682525, -0.007579192, -0.056257267, 0.038498487, 0.023768218, -0.02930198, -0.03665919, -0.0893647, -0.029508108]..., parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=c48b6326-7f0a-4f69-bf31-0cd97ac4d2fc, text=that you see in 19th century studio paintings, and a nude model sitting as close to it as possible without getting burned. Except hardly anyone else painted her besides me. The rest of the students spent their time chatting or occasionally trying to imitate things they'd seen in American art magazines.\n",
       "\n",
       "Our model turned out to live just down the street from me. She made a living from a combination ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=484, vector=[0.037977736, -0.023429813, -0.054210156, -0.04045854, 0.03319989, 0.014402444, 0.012993592, -0.028085148, -0.06652229, 0.0016825822]..., parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=e0e8043e-40e9-4f5c-8153-b13fb9a788f5, text=generic person, which you then modify to match the specific person you're painting. Whereas a still life you can, if you want, copy pixel by pixel from what you're seeing. You don't want to stop there, of course, or you get merely photographic accuracy, and what makes a still life interesting is that it's been through a head. You want to emphasize the visual cues that tell you, for example, that t ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=486, vector=[0.017387627, -0.011447118, -0.06267119, 0.015462681, 0.02985244, -0.044399984, 0.004035286, 0.023320246, -0.041054994, 0.010389974]..., parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=64f6da0c-a846-4f55-ac9d-aeffd8c7742a, text=to notice every leaf on every bush. But when you have to paint something, you have to look more closely, and when you do there's a lot to see. You can still be noticing new things after days of trying to paint something people usually take for granted, just as you can after days of trying to write an essay about something people usually take for granted.\n",
       "\n",
       "This is not the only way to paint. I'm not ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=479, vector=[-0.044350445, 0.062140558, -0.05280855, -0.0019838319, 0.07278342, -0.055867203, -0.020958003, 0.060267914, -0.073907, 0.05786469]..., parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=00483010-3e74-4944-b800-211297a36070, text=software for creating documents. You mean like Microsoft Word? Exactly. That was how I learned that low end software tends to eat high end software. But Interleaf still had a few years to live yet. [5]\n",
       "\n",
       "Interleaf had done something pretty bold. Inspired by Emacs, they'd added a scripting language, and even made the scripting language a dialect of Lisp. Now they wanted a Lisp hacker to write things ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=469, vector=[-0.058567457, 0.081068136, 0.024077367, 0.0144037185, 0.09939715, -0.0729219, -0.015725838, 0.07686363, -0.049698576, 0.04651235]..., parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=aac98a05-90ec-4f70-aeb1-414de7fc9691, text=this time gotten a contract to publish.\n",
       "\n",
       "The good part was that I got paid huge amounts of money, especially by art student standards. In Florence, after paying my part of the rent, my budget for everything else had been $7 a day. Now I was getting paid more than 4 times that every hour, even when I was just sitting in a meeting. By living cheaply I not only managed to save enough to go back to RI ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=479, vector=[-0.066288754, 0.020486224, 0.020769367, 0.02410046, 0.06145867, -0.04666862, -0.0289472, 0.037974462, -0.035975806, 0.034943167]..., parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=c92e0f39-4937-4543-a96d-84af47ccdd03, text=it.\n",
       "\n",
       "But the most important thing I learned, and which I used in both Viaweb and Y Combinator, is that the low end eats the high end: that it's good to be the \"entry level\" option, even though that will be less prestigious, because if you're not, someone else will be, and will squash you against the ceiling. Which in turn means that prestige is a danger sign.\n",
       "\n",
       "When I left to go back to RISD the ne ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=489, vector=[-0.004584177, 0.0276189, -0.049178105, 0.02639295, 0.090930454, -0.012189444, -0.013756909, 0.004262365, -0.06616627, 0.015455726]..., parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=c250b32a-8471-483a-8b69-c5954ddb6b86, text=it was more like the Accademia than not. Better organized, certainly, and a lot more expensive, but it was now becoming clear that art school did not bear the same relationship to art that medical school bore to medicine. At least not the painting department. The textile department, which my next door neighbor belonged to, seemed to be pretty rigorous. No doubt illustration and architecture were t ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=484, vector=[0.011460669, 0.049471214, -0.050929107, 0.009533009, 0.0056695887, -0.012837569, 0.002219239, 0.04522712, -0.10153424, 0.004576168]..., parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=749a0d64-21ec-4152-b788-8c0fa899742f, text=kids who \"could draw\" in high school, and now had come to what was supposed to be the best art school in the country, to learn to draw even better. They tended to be confused and demoralized by what they found at RISD, but they kept going, because painting was what they did. I was not one of the kids who could draw in high school, but at RISD I was definitely closer to their tribe than the tribe o ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=485, vector=[-0.020797122, 0.031381227, -0.047047637, 0.020442167, 0.023959447, -0.06757048, -0.0074096806, 0.09971002, -0.108874306, 0.00082083285]..., parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=0477d15a-937b-4a8b-ad82-8385220f17ee, text=Romans. You can do something similar on a map of New York City: if you zoom in on the Upper East Side, there's a tiny corner that's not rich, or at least wasn't in 1993. It's called Yorkville, and that was my new home. Now I was a New York artist — in the strictly technical sense of making paintings and living in New York.\n",
       "\n",
       "I was nervous about money, because I could sense that Interleaf was on the ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=486, vector=[-0.0731452, 0.008238716, -0.0652409, 0.05916067, 0.07764457, -0.07752297, -0.011856454, 0.10853216, -0.08457604, -0.008649131]..., parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=dddeb389-9797-4f13-b2e2-e7164f259662, text=photorealists, and I'd taken her painting class at Harvard. I've never known a teacher more beloved by her students. Large numbers of former students kept in touch with her, including me. After I moved to New York I became her de facto studio assistant.\n",
       "\n",
       "She liked to paint on big, square canvases, 4 to 5 feet on a side. One day in late 1994 as I was stretching one of these monsters there was somet ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=483, vector=[-0.02226281, 0.016266143, -0.09724324, 0.054839294, 0.05142105, -0.10997327, -0.021157773, 0.081448585, -0.100190006, -0.0007703028]..., parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=0ea85a23-39e0-46e5-9cfa-52d17d66b54f, text=I was right about that part. What I got wrong was the idea. I decided we should start a company to put art galleries online. I can't honestly say, after reading so many Y Combinator applications, that this was the worst startup idea ever, but it was up there. Art galleries didn't want to be online, and still don't, not the fancy ones. That's not how they sell. I wrote some software to generate web ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=472, vector=[-0.06592135, 0.040219873, -0.09019341, 0.015120991, 0.06592135, -0.10857964, 0.0021388705, 0.042938568, -0.100731865, 0.02292673]..., parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=10dda6e6-caa5-4961-a8c0-50f4dce70def, text=online stores. At first this was going to be normal desktop software, which in those days meant Windows software. That was an alarming prospect, because neither of us knew how to write Windows software or wanted to learn. We lived in the Unix world. But we decided we'd at least try writing a prototype store builder on Unix. Robert wrote a shopping cart, and I wrote a new site generator for stores  ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=469, vector=[-0.044771, 0.077939205, -0.043221936, 0.016963774, 0.0947663, -0.09500929, -0.009089364, 0.051574733, -0.09397658, 0.06463547]..., parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=3bc3fee5-cce0-4766-877f-79293ae114cd, text=is common now, but at the time it wasn't clear that it was even possible. To find out, we decided to try making a version of our store builder that you could control through the browser. A couple days later, on August 12, we had one that worked. The UI was horrible, but it proved you could build a whole store through the browser, without any client software or typing anything into the command line ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=474, vector=[-0.014542281, 0.08878622, -0.06298567, 0.04715505, 0.13122576, -0.072753504, -0.002707205, 0.025328998, -0.102865376, 0.06362563]..., parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=7d00af64-63ee-4057-ab6a-bc7609bd4585, text=later this deal became the model for Y Combinator's. We knew founders needed something like this, because we'd needed it ourselves.\n",
       "\n",
       "At this stage I had a negative net worth, because the thousand dollars or so I had in the bank was more than counterbalanced by what I owed the government in taxes. (Had I diligently set aside the proper proportion of the money I'd made consulting for Interleaf? No,  ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=488, vector=[-0.033374872, 0.10630641, -0.015908288, 0.04126908, 0.08253816, -0.04260476, 0.007697283, 0.114936955, -0.06216048, 0.012355038]..., parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=65c64559-a29f-4e53-93ab-5e14c94019cf, text=can make a guy running a store out of his bedroom look more legit than a big company.\n",
       "\n",
       "(If you're curious why my site looks so old-fashioned, it's because it's still made with this software. It may look clunky today, but in 1996 it was the last word in slick.)\n",
       "\n",
       "In September, Robert rebelled. \"We've been working on this for a month,\" he said, \"and it's still not done.\" This is funny in retrospect,  ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=496, vector=[-0.027819531, 0.09070735, 0.0051304507, -0.027231796, 0.10357225, -0.060667478, -0.007942607, 0.095996976, -0.12610215, 0.027444033]..., parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=51c402f2-7240-4196-b934-39b923acce7e, text=and if you could see inside Trevor's it would look like the worst excesses of Austrian Rococo.\n",
       "\n",
       "We opened for business, with 6 stores, in January 1996. It was just as well we waited a few months, because although we worried we were late, we were actually almost fatally early. There was a lot of talk in the press then about ecommerce, but not many people actually wanted online stores. [8]\n",
       "\n",
       "There we ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=481, vector=[-0.04253237, 0.07392454, 0.013313501, -0.024453698, 0.13677329, -0.055121437, 0.014834814, 0.08094351, -0.10786029, 0.035931963]..., parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=a8891cfb-3fce-41fc-881d-03fb5ef61258, text=a lot of startups making ecommerce software in the second half of the 90s. We were determined to be the Microsoft Word, not the Interleaf. Which meant being easy to use and inexpensive. It was lucky for us that we were poor, because that caused us to make Viaweb even more inexpensive than we realized. We charged $100 a month for a small store and $300 a month for a big one. This low price was a bi ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=476, vector=[-0.010524927, 0.06643315, -0.06027841, 0.025348656, 0.10615345, -0.10761282, -0.008795888, 0.104820974, -0.098285526, 0.03619877]..., parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=b9ce92fc-c13d-412a-befb-b8e82b917719, text=stores. But anything to get users.\n",
       "\n",
       "We learned a lot more about retail than we wanted to know. For example, that if you could only have a small image of a man's shirt (and all images were small then by present standards), it was better to have a closeup of the collar than a picture of the whole shirt. The reason I remember learning this was that it meant I had to rescan about 30 images of men's sh ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=488, vector=[0.0394093, 0.07387699, -0.0509603, 0.048365954, 0.13774723, -0.07505062, -0.0055245594, 0.1299642, -0.08382197, 0.030792374]..., parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=e12be00c-05c7-4391-8466-3d999fbb9c19, text=at the time is that growth rate is the ultimate test of a startup. Our growth rate was fine. We had about 70 stores at the end of 1996 and about 500 at the end of 1997. I mistakenly thought the thing that mattered was the absolute number of users. And that is the thing that matters in the sense that that's how much money you're making, and if you're not making enough, you might go out of business. ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=488, vector=[0.0011247798, -0.00021909054, -0.058989327, 0.075708225, 0.09722198, -0.07545587, 0.055077735, 0.10277392, 0.00029425658, 0.07236445]..., parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=a748daa5-6804-4719-8fdf-7c73abacfba7, text=mercy of investors for the entire life of the company. And since both we and our investors were noobs at startups, the result was a mess even by startup standards.\n",
       "\n",
       "It was a huge relief when Yahoo bought us. In principle our Viaweb stock was valuable. It was a share in a business that was profitable and growing rapidly. But it didn't feel very valuable to me; I had no idea how to value a business, ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=479, vector=[0.021928135, 0.019077936, -0.09935851, 0.08109272, 0.091574095, -0.07661822, 0.055961948, 0.07294054, -0.08060237, 0.08195085]..., parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=884cd7af-2f19-462e-a0c8-fe4f50359018, text=tried to continue my usual m.o. of programming till 3 in the morning, but fatigue combined with Yahoo's prematurely aged culture and grim cube farm in Santa Clara gradually dragged me down. After a few months it felt disconcertingly like working at Interleaf.\n",
       "\n",
       "Yahoo had given us a lot of options when they bought us. At the time I thought Yahoo was so overvalued that they'd never be worth anything, ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=477, vector=[0.02321774, 0.0148402825, -0.045890603, 0.09159958, 0.07628253, -0.043499205, 0.029892461, 0.10219438, -0.11242592, 0.08336591]..., parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=827af914-977b-4a13-b192-d09f8ff5cade, text=in me. Now I realize it was because he thought I was lying. My options at that point were worth about $2 million a month. If I was leaving that kind of money on the table, it could only be to go and start some new startup, and if I did, I might take people with me. This was the height of the Internet Bubble, and Yahoo was ground zero of it. My boss was at that moment a billionaire. Leaving then to ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=467, vector=[0.0074370974, 0.00995517, -0.06174548, 0.10106645, 0.033074588, -0.04197568, 0.022486972, 0.12592705, -0.090634994, 0.07951643]..., parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=e5d92d1e-327c-4fc5-b31c-ca87fe3f45a9, text=this problem by buying a house up in the Santa Cruz Mountains, with a beautiful view but miles from anywhere. I stuck it out for a few more months, then in desperation I went back to New York, where unless you understand about rent control you'll be surprised to hear I still had my apartment, sealed up like a tomb of my old life. Idelle was in New York at least, and there were other people trying  ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=478, vector=[-0.03939617, -0.013581642, 0.004814739, 0.06267002, 0.042156413, -0.11411088, 0.04278374, 0.07741222, -0.06655946, 0.0077435747]..., parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=f386916e-afeb-4d54-90a1-9cdbea3225fb, text=photograph it and print it, blown up, on canvas, and then use that as the underpainting for a second still life, painted from the same objects (which hopefully hadn't rotted yet).\n",
       "\n",
       "Meanwhile I looked for an apartment to buy. Now I could actually choose what neighborhood to live in. Where, I asked myself and various real estate agents, is the Cambridge of New York? Aided by occasional visits to act ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=492, vector=[-0.040362474, 0.024286918, -0.02630957, 0.013517051, 0.088030644, -0.08724573, -0.011758552, 0.04099644, -0.05781162, -0.007018904]..., parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=2592319e-ab0a-4d66-aadb-123fd089cd14, text=this idea would have to be embodied as one, so I decided to move to Cambridge and start it. I hoped to lure Robert into working on it with me, but there I ran into a hitch. Robert was now a postdoc at MIT, and though he'd made a lot of money the last time I'd lured him into working on one of my schemes, it had also been a huge time sink. So while he agreed that it sounded like a plausible idea, he ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=477, vector=[-0.038379394, 0.024685407, -0.013724181, -0.018797146, 0.052632008, -0.08491176, -0.050276704, 0.022616968, -0.04571707, -0.00016159695]..., parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=f87fccfd-a7e1-4c8c-8298-6367e13f1cb6, text=was replaced by \"software as a service,\" but it was current for long enough that I named this new company after it: it was going to be called Aspra.\n",
       "\n",
       "I started working on the application builder, Dan worked on network infrastructure, and the two undergrads worked on the first two services (images and phone calls). But about halfway through the summer I realized I really didn't want to run a compan ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=488, vector=[-0.031537034, 0.019595249, -0.10691024, -0.02616174, 0.09547472, -0.04898812, -0.092318036, 0.036897436, -0.06897051, 0.031030772]..., parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=86c6fc1b-642b-4c0f-a400-86c3fa94dc81, text=new Lisp, whose parentheses I now wouldn't even have to hide. A lot of Lisp hackers dream of building a new Lisp, partly because one of the distinctive features of the language is that it has dialects, and partly, I think, because we have in our minds a Platonic form of Lisp that all existing dialects fall short of. I certainly did. So at the end of the summer Dan and I switched to working on this ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=489, vector=[0.0039237197, 0.023974624, -0.08915746, -0.027403757, 0.12637967, -0.081244074, -0.07057566, 0.07086875, -0.08880576, 0.034672346]..., parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=999a9d5f-6c64-4802-8670-2c29c4999682, text=era there was a narrow channel to readers, guarded by fierce monsters known as editors. The only way to get an audience for anything you wrote was to get it published as a book, or in a newspaper or magazine. Now anyone could publish anything.\n",
       "\n",
       "This had been possible in principle since 1993, but not many people had realized it yet. I had been intimately involved with building the infrastructure of ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=480, vector=[0.024288623, 0.024016058, -0.016974779, 0.020033572, 0.16099541, -0.08728156, -0.05875303, 0.10957137, -0.083768494, 0.0407637]..., parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=25cf69f3-6e65-4a8a-b57a-5218af060329, text=point where I figured out what to work on, it was when I started publishing essays online. From then on I knew that whatever else I did, I'd always write essays too.\n",
       "\n",
       "I knew that online essays would be a marginal medium at first. Socially they'd seem more like rants posted by nutjobs on their GeoCities sites than the genteel and beautifully typeset compositions published in The New Yorker. But by  ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=490, vector=[0.042509366, -0.0092501845, -0.04260892, 0.014858368, 0.13910949, -0.07532886, -0.023511231, 0.10711962, -0.10904432, 0.0015617464]..., parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=37d729b2-a158-49ce-9900-3bd0bc12dbbe, text=current lack of prestige, it's a sign both that there's something real to be discovered there, and that you have the right kind of motives. Impure motives are a big danger for the ambitious. If anything is going to lead you astray, it will be the desire to impress people. So while working on things that aren't prestigious doesn't guarantee you're on the right track, it at least guarantees you're n ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=472, vector=[0.04529276, -0.009341214, 0.020823296, 0.04124518, 0.08416291, -0.05301996, 0.008789271, 0.13119513, -0.0527189, 0.014517768]..., parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=ff31768b-2324-4483-8146-2aa8ff453fec, text=would all invite their friends to one party. So for every guest, two thirds of the other guests would be people they didn't know but would probably like. One of the guests was someone I didn't know but would turn out to like a lot: a woman called Jessica Livingston. A couple days later I asked her out.\n",
       "\n",
       "Jessica was in charge of marketing at a Boston investment bank. This bank thought it understood ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=466, vector=[0.01819799, -0.03182732, -0.026231475, 0.062918745, 0.08346239, -0.08806171, -0.01436522, 0.17820846, -0.001847395, 0.07033899]..., parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=bc9adb4e-131a-4293-9893-59cd7e56425f, text=founders instead of MBAs, they should let the founders remain as CEO, and so on.\n",
       "\n",
       "One of my tricks for writing essays had always been to give talks. The prospect of having to stand up in front of a group of people and tell them something that won't waste their time is a great spur to the imagination. When the Harvard Computer Society, the undergrad computer club, asked me to give a talk, I decided ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=485, vector=[0.038747437, 0.017834086, -0.0074455705, 0.07993263, 0.13279338, -0.053790938, 0.009646765, 0.16397095, 0.0009131549, 0.030279456]..., parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=e176759f-36a7-41f4-8c74-e4efa8c9ed69, text=one angel investment.\n",
       "\n",
       "Meanwhile I had been scheming with Robert and Trevor about projects we could work on together. I missed working with them, and it seemed like there had to be something we could collaborate on.\n",
       "\n",
       "As Jessica and I were walking home from dinner on March 11, at the corner of Garden and Walker streets, these three threads converged. Screw the VCs who were taking so long to make up ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=476, vector=[0.018336639, 0.019985966, -0.043496966, 0.061607227, 0.13427465, -0.011852519, 0.03803155, 0.11525887, -0.011901028, 0.030561065]..., parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=5e6bc402-6f7a-4414-be12-e78311d1316b, text=were organized companies with people whose job it was to make investments, but they only did big, million dollar investments. And there were angels, who did smaller investments, but these were individuals who were usually focused on other things and made investments on the side. And neither of them helped founders enough in the beginning. We knew how helpless founders were in some respects, becaus ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=468, vector=[0.0064024017, 0.055864017, -0.03715842, 0.09555022, 0.09731967, -0.036368486, 0.0397178, 0.08373283, -0.006516942, 0.04101329]..., parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=b6b5c863-bf88-4eac-bdf4-a9b81b5a512a, text=firms were organized. It never occurred to us to try to raise a fund, and if it had, we wouldn't have known where to start. [14]\n",
       "\n",
       "The most distinctive thing about YC is the batch model: to fund a bunch of startups all at once, twice a year, and then to spend three months focusing intensively on trying to help them. That part we discovered by accident, not merely implicitly but explicitly due to ou ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=485, vector=[-0.013644166, 0.039866306, -0.032634743, 0.08430642, 0.093824886, -0.068112664, -0.032263894, 0.09821327, 0.0029262332, 0.013744604]..., parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=28caadf7-0152-4b80-90cb-04656ecaf991, text=— on tuesdays, since I was already cooking for the thursday diners on thursdays — and after dinner we'd bring in experts on startups to give talks.\n",
       "\n",
       "We knew undergrads were deciding then about summer jobs, so in a matter of days we cooked up something we called the Summer Founders Program, and I posted an announcement on my site, inviting undergrads to apply. I had never imagined that writing essa ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=489, vector=[0.0194155, 0.014942945, 0.0015044232, 0.04013384, 0.1173192, -0.035303798, 0.016301394, 0.107595555, 0.009818968, 0.047792003]..., parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=2746e235-b8c2-4d46-b966-e7976747fc9f, text=Sam Altman, who would later become the second president of YC. I don't think it was entirely luck that the first batch was so good. You had to be pretty bold to sign up for a weird thing like the Summer Founders Program instead of a summer job at a legit place like Microsoft or Goldman Sachs.\n",
       "\n",
       "The deal for startups was based on a combination of the deal we did with Julian ($10k for 10%) and what R ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=483, vector=[-0.012571322, 0.010419402, 0.006970003, 0.074874125, 0.14126717, -0.01348114, 0.03958898, 0.06405124, -0.03882948, 0.0021855428]..., parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=f9d8df31-620d-409e-85d1-a7952c65641b, text=you not only had colleagues, but colleagues who understood the problems you were facing and could tell you how they were solving them.\n",
       "\n",
       "As YC grew, we started to notice other advantages of scale. The alumni became a tight community, dedicated to helping one another, and especially the current batch, whose shoes they remembered being in. We also noticed that the startups were becoming one another's ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=480, vector=[-0.0061149863, 0.005913606, 0.008845236, 0.09313058, 0.15850168, -0.05743982, 0.033336163, 0.05520915, -0.056107614, 0.10416002]..., parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=5b421cab-dde2-41a7-8fe2-5072db9b9a3a, text=News in it. It was originally meant to be a news aggregator for startup founders and was called Startup News, but after a few months I got tired of reading about nothing but startups. Plus it wasn't startup founders we wanted to reach. It was future startup founders. So I changed the name to Hacker News and the topic to whatever engaged one's intellectual curiosity.\n",
       "\n",
       "HN was no doubt good for YC, b ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=480, vector=[-0.0348371, 0.01800779, -0.02318162, 0.07921704, 0.13635908, -0.049208842, 0.018252112, 0.07306593, -0.027521884, 0.14808641]..., parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=2a00678d-d1c4-495f-bfe6-f6054b245730, text=YC's internal software in Arc. But while I continued to work a good deal in Arc, I gradually stopped working on Arc, partly because I didn't have time to, and partly because it was a lot less attractive to mess around with the language now that we had all this infrastructure depending on it. So now my three projects were reduced to two: writing essays and working on YC.\n",
       "\n",
       "YC was different from othe ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=494, vector=[-0.015509707, 0.056496896, -0.07327663, 0.11170068, 0.12081407, -0.0060037575, 0.007458514, 0.07807963, -0.034852583, 0.10024735]..., parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=7c89f5e2-5c4a-4deb-922c-7c71c165fbc0, text=something Kevin Hale once said about companies: \"No one works harder than the boss.\" He meant it both descriptively and prescriptively, and it was the second part that scared me. I wanted YC to be good, so if how hard I worked set the upper bound on how hard everyone else worked, I'd better work very hard.\n",
       "\n",
       "One day in 2010, when he was visiting California for interviews, Robert Morris did somethin ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=488, vector=[0.046288613, 0.04556893, -0.0457325, 0.11979428, 0.074585184, -0.011457658, 0.008358121, 0.09061446, -0.04452212, 0.061369214]..., parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=b3223ae5-6458-4a70-a114-1d5a38336d7c, text=this set me thinking. It was true that on my current trajectory, YC would be the last thing I did, because it was only taking up more of my attention. It had already eaten Arc, and was in the process of eating essays too. Either YC was my life's work or I'd have to leave eventually. And it wasn't, so I would.\n",
       "\n",
       "In the summer of 2012 my mother had a stroke, and the cause turned out to be a blood clo ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=486, vector=[0.055007536, 0.05108489, -0.03557536, 0.10512685, 0.091729514, 0.04366204, 0.030898362, 0.08044437, -0.0019330336, 0.044205178]..., parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=89e0b36a-6fc1-45fe-80d1-a243250d6e7e, text=guard. Up till that point YC had been controlled by the original LLC we four had started. But we wanted YC to last for a long time, and to do that it couldn't be controlled by the founders. So if Sam said yes, we'd let him reorganize YC. Robert and I would retire, and Jessica and Trevor would become ordinary partners.\n",
       "\n",
       "When we asked Sam if he wanted to be president of YC, initially he said no. He  ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=497, vector=[0.031589653, 0.07312294, -0.029526517, 0.11695612, 0.0784668, 0.028139818, 0.00945322, 0.042581767, -0.026600922, 0.10058632]..., parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=641f6680-eaac-4f50-8e58-aa4ff4992f5f, text=only takes a few hours a week.)\n",
       "\n",
       "What should I do next? Rtm's advice hadn't included anything about that. I wanted to do something completely different, so I decided I'd paint. I wanted to see how good I could get if I really focused on it. So the day after I stopped working on YC, I started painting. I was rusty and it took a while to get back into shape, but it was at least completely engaging.  ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=484, vector=[-0.065560885, 0.08753525, -0.034742437, -0.004403174, 0.11319213, -0.032689888, 0.015680883, 0.10129941, -0.06628531, 0.08210202]..., parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=a36943d4-8855-46d5-bcd1-7f3447828976, text=one (or at least a good one) for you, then it's getting in the way of another project that is. And at 50 there was some opportunity cost to screwing around.\n",
       "\n",
       "I started writing essays again, and wrote a bunch of new ones over the next few months. I even wrote a couple that weren't about startups. Then in March 2015 I started working on Lisp again.\n",
       "\n",
       "The distinctive thing about Lisp is that its core  ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=494, vector=[-0.036554195, 0.022350991, -0.03752417, -0.009249403, 0.11024457, -0.054900575, 0.015561168, 0.052323215, -0.046725072, 0.03613849]..., parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=9e4f8598-c70b-44d0-a897-1917bbaae437, text=of computation gave it a power and elegance that other languages couldn't match. It was this that attracted me in college, though I didn't understand why at the time.\n",
       "\n",
       "McCarthy's 1960 Lisp did nothing more than interpret Lisp expressions. It was missing a lot of things you'd want in a programming language. So these had to be added, and when they were, they weren't defined using McCarthy's original ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=500, vector=[-0.0006533495, -0.011946962, -0.08585718, -0.022733273, 0.010020282, -0.04828306, -0.050604362, 0.0020794987, -0.09241873, 0.033117227]..., parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=90b1a388-7eaa-4a45-ac7b-77fe6641d5aa, text=I decided to take a shot at it. It took 4 years, from March 26, 2015 to October 12, 2019. It was fortunate that I had a precisely defined goal, or it would have been hard to keep at it for so long.\n",
       "\n",
       "I wrote this new Lisp, called Bel, in itself in Arc. That may sound like a contradiction, but it's an indication of the sort of trickery I had to engage in to make this work. By means of an egregious c ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=471, vector=[0.0483754, 0.040378466, -0.03810661, 0.053736985, -0.012942011, -0.08021169, -0.029912781, 0.03638, -0.06936736, 0.029079767]..., parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=fed50714-a94e-4061-bc15-4e9d2fc70e6c, text=more essays till Bel was done. But I told few people about Bel while I was working on it. So for years it must have seemed that I was doing nothing, when in fact I was working harder than I'd ever worked on anything. Occasionally after wrestling for hours with some gruesome bug I'd check Twitter or HN and see someone asking \"Does Paul Graham still code?\"\n",
       "\n",
       "Working on Bel was hard but satisfying. I  ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=471, vector=[0.06559414, 0.0266457, -0.07093248, 0.04206247, 0.038472895, -0.08860424, -0.03604916, 0.14063776, -0.10050813, 0.008981611]..., parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=49cb3211-1cdf-4a38-ae26-e622b88a2e4e, text=by birth, that seemed the obvious choice. We only meant to stay for a year, but we liked it so much that we still live there. So most of Bel was written in England.\n",
       "\n",
       "In the fall of 2019, Bel was finally finished. Like McCarthy's original Lisp, it's a spec rather than an implementation, although like McCarthy's Lisp it's a spec expressed as code.\n",
       "\n",
       "Now that I could write essays again, I wrote a bunc ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=510, vector=[0.032647964, 0.006369719, -0.03749467, -0.012217743, 0.10218476, -0.03810051, -0.084144235, 0.05762197, -0.12668756, -0.023341611]..., parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=602759c1-fe65-4dc5-8fda-7ae6d4249527, text=OSes. I went straight from batch processing to microcomputers, which made microcomputers seem all the more exciting.\n",
       "\n",
       "[2] Italian words for abstract concepts can nearly always be predicted from their English cognates (except for occasional traps like polluzione). It's the everyday words that differ. So if you string together a lot of abstract concepts with a few simple verbs, you can make a little ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=531, vector=[-0.03932356, 0.05249898, -0.021654986, 0.016595962, 0.13918649, -0.06783652, -0.053748958, 0.03746549, -0.037735753, 0.0060176197]..., parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=2a7916b5-4794-43f1-90f4-ff55d1b0e0ff, text=In the 1990s the exponential growth in the power of commodity (i.e. Intel) processors rolled up high-end, special-purpose hardware and software companies like a bulldozer.\n",
       "\n",
       "[6] The signature style seekers at RISD weren't specifically mercenary. In the art world, money and coolness are tightly coupled. Anything expensive comes to be seen as cool, and anything seen as cool will soon become equally e ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=509, vector=[0.04108548, 0.033236522, -0.03504248, 0.09606293, 0.18726367, -0.072238214, -0.041154943, 0.11203869, -0.045322534, 0.083421245]..., parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=5041ad05-7619-4ed2-a969-8845cd0e2a99, text=were generated, not when shoppers visited them.\n",
       "\n",
       "[10] This was the first instance of what is now a familiar experience, and so was what happened next, when I read the comments and found they were full of angry people. How could I claim that Lisp was better than other languages? Weren't they all Turing complete? People who see the responses to essays I write sometimes tell me how sorry they feel fo ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=499, vector=[0.048909534, 0.011038376, -0.05572314, 0.09309676, 0.15192313, -0.04789761, -0.0007225751, 0.07521949, -0.044592, 0.07150911]..., parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=173d166b-8fb1-497a-bd47-c94542aa6f7d, text=once been much more expensive to start, and proportionally rare. Now they could be cheap and common, but the VCs' customs still reflected the old world, just as customs about writing essays still reflected the constraints of the print era.\n",
       "\n",
       "Which in turn implies that people who are independent-minded (i.e. less influenced by custom) will have an advantage in fields affected by rapid change (where  ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=502, vector=[0.04851072, -0.016810644, -0.035268046, 0.114381, 0.16000989, 0.026107961, 0.007847827, 0.07492745, -0.04219815, 0.09352208]..., parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=ac74b894-771d-4e1b-b9e0-0e9e0692e02e, text=the Viaweb logo had been a white V on a red circle, so I made the YC logo a white Y on an orange square.\n",
       "\n",
       "[14] YC did become a fund for a couple years starting in 2009, because it was getting so big I could no longer afford to fund it personally. But after Heroku got bought we had enough money to go back to being self-funded.\n",
       "\n",
       "[15] I've never liked the term \"deal flow,\" because it implies that the ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=498, vector=[0.0049796477, 0.035638493, -0.052701972, 0.085854836, 0.1597518, 0.0023407703, -0.00250242, -0.005235768, -0.045984067, 0.13583606]..., parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=a7c0300e-257b-4e92-b3bc-2a9a9eb016cb, text=misinterpretations of them on forums. Individually these two phenomena are tedious but bearable, but the combination is disastrous. You actually have to respond to the misinterpretations, because the assumption that you're present in the conversation means that not responding to any sufficiently upvoted misinterpretation reads as a tacit admission that it's correct. But that in turn encourages mor ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=316, vector=[0.022566214, 0.066998914, -0.07452098, 0.060141586, 0.07997886, -0.013242344, 0.011370573, 0.014239456, -0.07147717, 0.06224077]..., parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5)]}, mapped_documents={}, transformer_setups={})"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# construct the db\n",
    "\n",
    "restored_db = LocalDB.load_state(\"developer_notes/db_states.pkl\")\n",
    "restored_db"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1 68 ['Sequential__0_1_1.vectorizer_1.vectorizer.model_client_1.batch_embedder_']\n"
     ]
    }
   ],
   "source": [
    "len_documents=len(restored_db.documents)\n",
    "keys = list(restored_db.transformed_documents.keys())\n",
    "len_transformed_documents=len(restored_db.transformed_documents[keys[0]])\n",
    "print(len_documents, len_transformed_documents, keys)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[-0.07920883,\n",
       " 0.038013875,\n",
       " -0.00059247564,\n",
       " -0.01969087,\n",
       " 0.1272431,\n",
       " -0.03413296,\n",
       " 0.012310769,\n",
       " 0.11960851,\n",
       " -0.07208321,\n",
       " 0.089897245]"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# lets' print out part of the vector\n",
    "restored_db.transformed_documents[keys[0]][0].vector[0:10]\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we have prepared the embeddings which can be used in ``FAISSRetriever``. The ``FAISSRetriever`` is a simple wrapper around the FAISS library. It is a simple and efficient way to search for the nearest neighbors in the embedding space."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "FAISSRetriever(\n",
      "  top_k=5, metric=prob\n",
      "  (embedder): Embedder(\n",
      "    model_kwargs={'model': 'text-embedding-3-small', 'dimensions': 256, 'encoding_format': 'float'}, \n",
      "    (model_client): OpenAIClient()\n",
      "  )\n",
      ")\n"
     ]
    }
   ],
   "source": [
    "\n",
    "from lightrag.components.retriever import FAISSRetriever\n",
    "\n",
    "\n",
    "\n",
    "retriever = FAISSRetriever(embedder=components[\"embedder\"], top_k=5)\n",
    "print(retriever)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "68 <class 'list'> [-0.07920883, 0.038013875, -0.00059247564, -0.01969087, 0.1272431, -0.03413296, 0.012310769, 0.11960851, -0.07208321, 0.089897245]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{256}"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "documents = restored_db.transformed_documents[keys[0]]\n",
    "vectors = [doc.vector for doc in documents]\n",
    "print(len(vectors), type(vectors), vectors[0][0:10])\n",
    "\n",
    "# check if all embeddings are the same length\n",
    "dimensions = set([len(vector) for vector in vectors])\n",
    "dimensions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# convert vectors to numpy array\n",
    "import numpy as np\n",
    "vectors_np = np.array(vectors, dtype=np.float32)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "retriever.build_index_from_documents(documents=vectors)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[RetrieverOutput(doc_indices=[24, 25, 17, 32, 38], doc_scores=[0.7670000195503235, 0.7459999918937683, 0.734000027179718, 0.7329999804496765, 0.7200000286102295], query='What happened at Viaweb and Interleaf?', documents=None),\n",
       " RetrieverOutput(doc_indices=[47, 44, 49, 45, 46], doc_scores=[0.800000011920929, 0.7900000214576721, 0.7879999876022339, 0.7799999713897705, 0.7749999761581421], query='What company did Paul Graham co-found?', documents=None)]"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# retriever for a single query\n",
    "query = \"What happened at Viaweb and Interleaf?\"\n",
    "second_query = \"What company did Paul Graham co-found?\"\n",
    "\n",
    "output = retriever(input=[query, second_query])\n",
    "output"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[RetrieverOutput(doc_indices=[24, 25, 17, 32, 38], doc_scores=[0.7670000195503235, 0.7459999918937683, 0.734000027179718, 0.7329999804496765, 0.7200000286102295], query='What happened at Viaweb and Interleaf?', documents=[Document(id=10dda6e6-caa5-4961-a8c0-50f4dce70def, text=online stores. At first this was going to be normal desktop software, which in those days meant Windows software. That was an alarming prospect, because neither of us knew how to write Windows software or wanted to learn. We lived in the Unix world. But we decided we'd at least try writing a prototype store builder on Unix. Robert wrote a shopping cart, and I wrote a new site generator for stores  ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=469, vector=[-0.044771, 0.077939205, -0.043221936, 0.016963774, 0.0947663, -0.09500929, -0.009089364, 0.051574733, -0.09397658, 0.06463547]..., parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=3bc3fee5-cce0-4766-877f-79293ae114cd, text=is common now, but at the time it wasn't clear that it was even possible. To find out, we decided to try making a version of our store builder that you could control through the browser. A couple days later, on August 12, we had one that worked. The UI was horrible, but it proved you could build a whole store through the browser, without any client software or typing anything into the command line ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=474, vector=[-0.014542281, 0.08878622, -0.06298567, 0.04715505, 0.13122576, -0.072753504, -0.002707205, 0.025328998, -0.102865376, 0.06362563]..., parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=aac98a05-90ec-4f70-aeb1-414de7fc9691, text=this time gotten a contract to publish.\n",
       " \n",
       " The good part was that I got paid huge amounts of money, especially by art student standards. In Florence, after paying my part of the rent, my budget for everything else had been $7 a day. Now I was getting paid more than 4 times that every hour, even when I was just sitting in a meeting. By living cheaply I not only managed to save enough to go back to RI ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=479, vector=[-0.066288754, 0.020486224, 0.020769367, 0.02410046, 0.06145867, -0.04666862, -0.0289472, 0.037974462, -0.035975806, 0.034943167]..., parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=a748daa5-6804-4719-8fdf-7c73abacfba7, text=mercy of investors for the entire life of the company. And since both we and our investors were noobs at startups, the result was a mess even by startup standards.\n",
       " \n",
       " It was a huge relief when Yahoo bought us. In principle our Viaweb stock was valuable. It was a share in a business that was profitable and growing rapidly. But it didn't feel very valuable to me; I had no idea how to value a business, ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=479, vector=[0.021928135, 0.019077936, -0.09935851, 0.08109272, 0.091574095, -0.07661822, 0.055961948, 0.07294054, -0.08060237, 0.08195085]..., parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=f87fccfd-a7e1-4c8c-8298-6367e13f1cb6, text=was replaced by \"software as a service,\" but it was current for long enough that I named this new company after it: it was going to be called Aspra.\n",
       " \n",
       " I started working on the application builder, Dan worked on network infrastructure, and the two undergrads worked on the first two services (images and phone calls). But about halfway through the summer I realized I really didn't want to run a compan ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=488, vector=[-0.031537034, 0.019595249, -0.10691024, -0.02616174, 0.09547472, -0.04898812, -0.092318036, 0.036897436, -0.06897051, 0.031030772]..., parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5)]),\n",
       " RetrieverOutput(doc_indices=[47, 44, 49, 45, 46], doc_scores=[0.800000011920929, 0.7900000214576721, 0.7879999876022339, 0.7799999713897705, 0.7749999761581421], query='What company did Paul Graham co-found?', documents=[Document(id=b6b5c863-bf88-4eac-bdf4-a9b81b5a512a, text=firms were organized. It never occurred to us to try to raise a fund, and if it had, we wouldn't have known where to start. [14]\n",
       " \n",
       " The most distinctive thing about YC is the batch model: to fund a bunch of startups all at once, twice a year, and then to spend three months focusing intensively on trying to help them. That part we discovered by accident, not merely implicitly but explicitly due to ou ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=485, vector=[-0.013644166, 0.039866306, -0.032634743, 0.08430642, 0.093824886, -0.068112664, -0.032263894, 0.09821327, 0.0029262332, 0.013744604]..., parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=bc9adb4e-131a-4293-9893-59cd7e56425f, text=founders instead of MBAs, they should let the founders remain as CEO, and so on.\n",
       " \n",
       " One of my tricks for writing essays had always been to give talks. The prospect of having to stand up in front of a group of people and tell them something that won't waste their time is a great spur to the imagination. When the Harvard Computer Society, the undergrad computer club, asked me to give a talk, I decided ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=485, vector=[0.038747437, 0.017834086, -0.0074455705, 0.07993263, 0.13279338, -0.053790938, 0.009646765, 0.16397095, 0.0009131549, 0.030279456]..., parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=2746e235-b8c2-4d46-b966-e7976747fc9f, text=Sam Altman, who would later become the second president of YC. I don't think it was entirely luck that the first batch was so good. You had to be pretty bold to sign up for a weird thing like the Summer Founders Program instead of a summer job at a legit place like Microsoft or Goldman Sachs.\n",
       " \n",
       " The deal for startups was based on a combination of the deal we did with Julian ($10k for 10%) and what R ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=483, vector=[-0.012571322, 0.010419402, 0.006970003, 0.074874125, 0.14126717, -0.01348114, 0.03958898, 0.06405124, -0.03882948, 0.0021855428]..., parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=e176759f-36a7-41f4-8c74-e4efa8c9ed69, text=one angel investment.\n",
       " \n",
       " Meanwhile I had been scheming with Robert and Trevor about projects we could work on together. I missed working with them, and it seemed like there had to be something we could collaborate on.\n",
       " \n",
       " As Jessica and I were walking home from dinner on March 11, at the corner of Garden and Walker streets, these three threads converged. Screw the VCs who were taking so long to make up ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=476, vector=[0.018336639, 0.019985966, -0.043496966, 0.061607227, 0.13427465, -0.011852519, 0.03803155, 0.11525887, -0.011901028, 0.030561065]..., parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5), Document(id=5e6bc402-6f7a-4414-be12-e78311d1316b, text=were organized companies with people whose job it was to make investments, but they only did big, million dollar investments. And there were angels, who did smaller investments, but these were individuals who were usually focused on other things and made investments on the side. And neither of them helped founders enough in the beginning. We knew how helpless founders were in some respects, becaus ..., meta_data={'title': \"Paul Graham's essay\", 'path': 'data/paul_graham/paul_graham_essay.txt', 'size': 75042, 'creation_date': 'Sun Jun 16 13:13:07 2024', 'last_modified_date': 'Sun Jun 16 13:13:07 2024'}, estimated_num_tokens=468, vector=[0.0064024017, 0.055864017, -0.03715842, 0.09555022, 0.09731967, -0.036368486, 0.0397178, 0.08373283, -0.006516942, 0.04101329]..., parent_doc_id=024146ce-84e5-43a2-a1fe-e5385ad9aab5)])]"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# get initial documents\n",
    "for per_query_output in output:\n",
    "    per_query_output.documents = [documents[idx] for idx in per_query_output.doc_indices]\n",
    "\n",
    "output"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the RAG notes, we will combine this with Generator to get the end to end response."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## BM25Retriever\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[RetrieverOutput(doc_indices=[38, 17, 39, 25, 63], doc_scores=[10.1114636573896, 9.649273483751845, 9.355255256567045, 9.283456223615227, 9.064007693239802], query='What happened at Viaweb and Interleaf?', documents=None),\n",
       " RetrieverOutput(doc_indices=[60, 59, 38, 39, 14], doc_scores=[14.466181996644409, 14.269841646528711, 12.763172741824395, 10.693920475917215, 5.3626785714999], query='What company did Paul Graham co-found?', documents=None)]"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from lightrag.components.retriever import BM25Retriever\n",
    "\n",
    "index_strings = [doc.text for doc in documents]\n",
    "\n",
    "retriever = BM25Retriever(documents=index_strings)\n",
    "\n",
    "# retriever.build_index_from_documents(documents=index_strings)\n",
    "\n",
    "output = retriever(input=[query, second_query])\n",
    "output"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "ename": "AttributeError",
     "evalue": "'InMemoryBM25Retriever' object has no attribute 'retrieve'",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mAttributeError\u001b[0m                            Traceback (most recent call last)",
      "Cell \u001b[0;32mIn[48], line 3\u001b[0m\n\u001b[1;32m      1\u001b[0m retriever \u001b[38;5;241m=\u001b[39m InMemoryBM25Retriever(top_k\u001b[38;5;241m=\u001b[39m\u001b[38;5;241m1\u001b[39m)\n\u001b[1;32m      2\u001b[0m retriever\u001b[38;5;241m.\u001b[39mbuild_index_from_documents([\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mhello world\u001b[39m\u001b[38;5;124m\"\u001b[39m, \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mworld is beautiful\u001b[39m\u001b[38;5;124m\"\u001b[39m, \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mtoday is a good day\u001b[39m\u001b[38;5;124m\"\u001b[39m])\n\u001b[0;32m----> 3\u001b[0m output \u001b[38;5;241m=\u001b[39m \u001b[43mretriever\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mretrieve\u001b[49m(\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mhello\u001b[39m\u001b[38;5;124m\"\u001b[39m)\n\u001b[1;32m      4\u001b[0m output\n",
      "File \u001b[0;32m~/Documents/test/LightRAG/lightrag/core/component.py:769\u001b[0m, in \u001b[0;36mComponent.__getattr__\u001b[0;34m(self, name)\u001b[0m\n\u001b[1;32m    766\u001b[0m     \u001b[38;5;28;01mif\u001b[39;00m name \u001b[38;5;129;01min\u001b[39;00m components:\n\u001b[1;32m    767\u001b[0m         \u001b[38;5;28;01mreturn\u001b[39;00m components[name]\n\u001b[0;32m--> 769\u001b[0m \u001b[38;5;28;01mraise\u001b[39;00m \u001b[38;5;167;01mAttributeError\u001b[39;00m(\n\u001b[1;32m    770\u001b[0m     \u001b[38;5;124mf\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124m'\u001b[39m\u001b[38;5;132;01m{\u001b[39;00m\u001b[38;5;28mtype\u001b[39m(\u001b[38;5;28mself\u001b[39m)\u001b[38;5;241m.\u001b[39m\u001b[38;5;18m__name__\u001b[39m\u001b[38;5;132;01m}\u001b[39;00m\u001b[38;5;124m'\u001b[39m\u001b[38;5;124m object has no attribute \u001b[39m\u001b[38;5;124m'\u001b[39m\u001b[38;5;132;01m{\u001b[39;00mname\u001b[38;5;132;01m}\u001b[39;00m\u001b[38;5;124m'\u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[1;32m    771\u001b[0m )\n",
      "\u001b[0;31mAttributeError\u001b[0m: 'InMemoryBM25Retriever' object has no attribute 'retrieve'"
     ]
    }
   ],
   "source": [
    "retriever = BM25Retriever(top_k=1)\n",
    "retriever.build_index_from_documents([\"hello world\", \"world is beautiful\", \"today is a good day\"])\n",
    "output = retriever.retrieve(\"hello\")\n",
    "output"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# save the index\n",
    "\n",
    "path = \"developer_notes/bm25_index.json\"\n",
    "retriever.save_to_file(path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "retriever_loaded = BM25Retriever.load_from_file(path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[RetrieverOutput(doc_indices=[0], doc_scores=[0.6229580777634034], query='hello', documents=None)]"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# test the loaded index\n",
    "output = retriever_loaded.retrieve(\"hello\", top_k=1)\n",
    "output"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "InMemoryBM25Retriever(\n",
       "  top_k=1, k1=1.5, b=0.75, epsilon=0.25\n",
       "  Number of documents: 3\n",
       ")"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "retriever_loaded"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "lightrag-project",
   "language": "python",
   "name": "light-rag-project"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
